7. A Scikit-learn Transformer

Duration1h15
Learning Objectives

Apply:
[BC07] multiple data-preparation and modeling steps chained into one pipeline that automatically keeps each fold’s training data separate from its evaluation data
[BC04] test-driven development to build a reusable, standards-compliant data-preparation component

Analyze:
[BC07] why a data-preparation step must be learned separately for each evaluation fold, or the result becomes overly optimistic and misleading

Prerequisites

Introduction

Several steps in this pipeline learn a decision from training data — Module 2’s missingness threshold decides which columns to drop, Module 4’s RFE decides which features to keep — and, until now, each has been computed once, on the whole 4-city dataset, before evaluate_group_cv ever splits into folds. This module teaches the tool scikit-learn itself provides to make “learn only from training data” a structural guarantee instead of a discipline you have to remember: a real transformer, BaseEstimator/TransformerMixin, and Pipeline to chain several of them together. This is scikit-learn’s own documented recommended practice, not a convention specific to this course — and it is genuinely new to this course.

Notebook

Files: notebooks/session3/07_sklearn_transformer.ipynb

Open it and select the same .venv kernel as your other notebooks. It is entirely self-contained — every example runs on toy data, no air_quality code needed until the “Now do it for real” section. Work through the cells in order:

  1. See why a plain function can’t keep the “learned once, reused everywhere” promise, using a tiny synthetic example
  2. Build a small illustrative transformer (ColumnMeanImputer) step by step: __init__, then fit, then transform — each tested immediately after you write it
  3. Compose it into a real sklearn.pipeline.Pipeline, evaluated with cross_val_score + GroupKFold on synthetic grouped data — no hand-written per-fold loop
  4. Chain a second, ready-made transformer (StandardScaler) after it, in the same Pipeline — the point being that your own transformer and scikit-learn’s built-in ones compose for free, because they share the same contract
  5. Once AirQualityCleaner is complete (see Implementation below), you will chain it with RFE — Module 2’s cleaning and Module 4’s feature selection have the exact same structural gap you just saw generically, and this fixes both at once
A threshold can look safe by accident

If you do the optional check suggested in the Introduction — recomputing columns_above_missing_threshold on all four cities vs. excluding one at a time — you will find that at 0.7, nothing changes: the one column crossing that threshold (uvaerosollayerheight_aerosol_height, 94.7% missing) is so far above it that no single city’s absence moves the average enough to matter. That does not mean the process was safe at 0.7 — it means that particular number happened not to expose it. At 0.6, the same calculation genuinely depends on which cities are included, which is why the Implementation below asks you to use missing_threshold=0.6, not Module 2’s 0.7.

Implementation

Files: src/air_quality/transformers.py, src/air_quality/workflows.py

  • Complete AirQualityCleaner’s fit/transform in the already-scaffolded src/air_quality/transformers.py — its docstring and tests/test_transformers.py specify exactly what each should do. RFE — the tool behind Module 4’s select_features_rfe — is already a proper scikit-learn transformer itself, so there is no second class to write for feature selection.
  • Update run_advanced in src/air_quality/workflows.py once more: rebuild on Module 4’s wider column scope (SATELLITE_COLUMNS/SCOPE_COLUMNS_MODULE_4, not Module 2/3’s narrower DEFAULT_COLUMNS), then replace the current sequence with a single Pipeline chaining AirQualityCleaner (with missing_threshold=0.6, not Module 2’s 0.7), RFE (the same estimator and n_features_to_select Module 4 used), and a model, evaluated with cross_val_score + GroupKFold. Return the per-fold scores alongside the aggregate, the way evaluate_group_cv’s "folds" key already does — you will need them for the Reflection question below. Verify it with uv run python scripts/run_pipeline.py, never from the notebook.

Reflection

Files: reflection/session_3/module_7.md

Details

Question:

  1. Suppose ColumnMeanImputer.fit computed its means from outside X — a global variable, a file — instead of from X itself. Would clone() still protect you against leakage between folds? Why or why not?
  2. Module 2’s missingness threshold and Module 4’s RFE selection are each computed once on the whole dataset, before any fold split happens. Why does that make them exactly the same structural problem as the badly written fit above?
  3. Look at your own run’s results per fold, not just the mean — is every fold in the same range, or does one stand out? If one does, what is different about that fold’s training cities that could explain it?
  4. Try swapping the model for Ridge, both as RFE’s estimator and as the final model, and rerun: does anything change, and why might a regularized model behave differently here than a plain LinearRegression?