1. Correlation Analysis

Duration1h30
Learning Objectives

Apply:
[BC07] statistical significance testing to separate a real relationship in the data from noise

Analyze:
[BC07] why this dataset’s specific pattern of missing values makes pairwise handling — not a single global drop of incomplete rows — the difference between a trustworthy and a misleading result

Prerequisites

Introduction

In Understand the Air Quality Problem, you built a first correlation heatmap on two cities — a quick way to see which variables might matter, but not a rigorous one. This module does it properly: Pearson and Spearman correlation, their p-values, and the correct way to handle missing data when comparing many variables at once — on the full dataset, all four cities.

This activity produces an analysis notebook, not a piece of the production pipeline: correlation screening informs decisions (which variables look worth investigating further), it is not itself a step workflows.py needs to run.

Notebook

Files: notebooks/session3/01_correlation_analysis.ipynb

Open it and select the same .venv kernel as your other notebooks. Complete each # TODO cell in order:

  1. Define correlate_with_target, a function computing Pearson and Spearman correlation plus their p-values between one column and pm2_5
  2. Apply it to every satellite column, building a sorted correlation_table
  3. Define pairwise_correlation_matrices, generalizing the same idea to every pair of columns at once
  4. Apply it, and build a mask that hides non-significant cells and the redundant upper triangle
  5. Draw the masked correlation heatmap
  6. Zoom in on the most correlated variable with a scatter plot and regression line
A common bug to avoid

Compute p-values one pair of columns at a time, dropping missing values only for that pair — not with a single dropna() applied across every column at once. uvaerosollayerheight_aerosol_height alone is missing over 90% of the time: requiring every column to be present simultaneously collapses the usable sample to a handful of rows, almost all from one city, and produces misleading correlations for every variable in the table, not just that one.

Reflection

Files: reflection/session_3/module_1.md

Details

Question: Compare correlation_table to what you found in the Session 2 discovery notebook, which only used Kampala and Nairobi. Did any variable’s apparent importance change now that Lagos and Bujumbura are included? Does the most correlated variable have a large enough n for you to trust it? What would you investigate next?