4. Feature Selection

Duration1h15
Learning Objectives

Apply:
[BC07] a pipeline extended to automatically select useful features out of a large, noisy set of candidates
[BC04] test-driven development to implement and fairly compare two competing feature-selection strategies

Analyze:
[BC07] why a method that judges each feature in isolation can be misled by redundant information, and why a more holistic method resists that trap

Prerequisites

Introduction

Every activity so far has used the same 8 hand-picked satellite columns (DEFAULT_COLUMNS). The raw data actually has 70 satellite columns — the 8 you have been using, plus secondary measurements (viewing angles, alternate retrievals, sensor geometry) that were deliberately set aside until now. This module uses all 70 and lets an algorithm choose instead of a human — and shows that simply using more columns can make a linear model considerably worse, not better.

Notebook

Files: notebooks/session3/04_feature_selection.ipynb

Open it and select the same .venv kernel as your other notebooks. Work through the cells in order:

  1. Rebuild the cleaned dataset, scoped to all 70 satellite columns instead of the 8-column DEFAULT_COLUMNS
  2. Fit a plain LinearRegression on every candidate feature, and look at the fitted coefficients
  3. Diagnose why before touching any selection method: compute the correlation of every candidate feature against every other one (not against pm2_5), and look at the strongest pairs
  4. Once select_k_best_features is complete (see Implementation below), apply it with k=8
  5. Once select_features_rfe is complete, apply it with n_features_to_select=8, and compare its picks to select_k_best_features’s
  6. Measure the difference: evaluate all three feature sets (every candidate, SelectKBest’s 8, RFE’s 8) with evaluate_group_cv (Module 3)
More features is not automatically better

Fitting on every candidate feature produces coefficients in the hundreds of thousands, for a target that never exceeds ~456 in this data — the same family of problem as site_latitude/site_longitude in Session 2 (a linear model producing a huge, unstable coefficient), but from a different cause here: dozens of pairs of columns carry almost the same information (multicollinearity — the correlation step in Step 3 shows you exactly which ones), so the model cannot uniquely decide how much credit to assign each one.

Pearson, not Spearman — and no p-value

For the correlation-among-features step, use Pearson only: the question is whether two columns confuse a linear model’s coefficients, a linear-algebra fact specific to linear correlation — not whether a relationship exists at all (Module 1’s question, where Spearman also made sense). Skip the p-value too: with ~8000 rows, even a trivial correlation (r=0.1) has a p-value around 10⁻¹⁶ — large samples make p-values stop being informative about anything except sample size (see Lin, Lucas & Shmueli, 2013). Only the size of r tells you anything useful here.

Implementation

Files: src/air_quality/selection.py, src/air_quality/workflows.py

  • Complete select_k_best_features and select_features_rfe in the new src/air_quality/selection.py (already scaffolded) — their docstrings and tests/test_selection.py specify exactly what each should do
  • Update run_advanced in src/air_quality/workflows.py again: scope to all 70 satellite columns, add a feature-selection step before evaluating (select_features_rfe is the safer default, given what you saw in the notebook). As with the previous modules, verify it with uv run python scripts/run_pipeline.py, never from the notebook.

Reflection

Files: reflection/session_3/module_4.md

Details

Question: RFE’s 8 columns come close to matching (or beating) the RMSE you saw in Module 3 with the 8 hand-picked DEFAULT_COLUMNS — an algorithm rediscovered, on its own, roughly what a human chose by inspection. Why does SelectKBest do noticeably worse here despite also picking 8 columns? What does this tell you about when a univariate method is enough, and when it is not? Cite one concrete number from your own run showing why the all-features model should not be trusted.