2. Advanced Cleaning

Duration1h
Learning Objectives

Apply:
[BC07] a production pipeline extended to handle a more realistic, imbalanced dataset without breaking
[BC04] test-driven development to harden a data-cleaning step against a real edge case before it causes a silent failure

Analyze:
[BC07] why a cleaning strategy that works on balanced data can quietly fail once the data becomes imbalanced

Prerequisites

Introduction

Session 2’s fill_missing_by_city (forward/backward-fill within each city) worked well enough on Kampala and Nairobi — it never had to face a city with zero data for some column. The full dataset has two smaller cities, Lagos and Bujumbura, and Bujumbura is small enough (123 rows) that this actually happens. This module finds exactly where the Session 2 cleaning strategy breaks, and fixes it with a missingness check applied before filling — a small, targeted extension of src/air_quality/data.py, not a rewrite of anything from Session 2.

Notebook

Files: notebooks/session3/02_advanced_cleaning.ipynb

Open it and select the same .venv kernel as your other notebooks. Work through the cells in order:

  1. Scope the full dataset to all four cities, uncapped, and look at the real row-count imbalance
  2. Reapply Session 2’s per-city missing-value heatmap — this time it reveals a column entirely missing in Bujumbura
  3. Watch fill_missing_by_city fail to fill it, and see the exact error this causes downstream
  4. Once columns_above_missing_threshold/drop_columns are complete (see Implementation below), apply them before filling and confirm the fix

Implementation

Files: src/air_quality/data.py, src/air_quality/workflows.py, scripts/run_pipeline.py

  • Complete columns_above_missing_threshold and drop_columns in src/air_quality/data.py (already scaffolded, in the “Session 3 — optional modules” section at the bottom of the file) — their docstrings and tests/test_data_advanced.py specify exactly what they should do
  • Write run_advanced yourself in src/air_quality/workflows.py (also already scaffolded, with AdvancedPipelineConfig): scope to all four cities, drop high-missing columns, fill the rest per city, add temporal features, then evaluate with evaluate_manual_split using config.train_city/config.test_city — the same sequence you ran in the notebook, packaged the way run_baseline packages Session 2’s pipeline
  • Update scripts/run_pipeline.py: replace the run_baseline/PipelineConfig import and call with run_advanced/AdvancedPipelineConfig. This script — never the notebook — is how you confirm run_advanced works, exactly like it already confirmed run_baseline for you in Session 2.
run_advanced is yours to maintain, not given to you

Unlike run_baseline, run_advanced is not handed to you complete — you write it, and you will keep extending it as you complete more optional modules. There is no test for it: its correct shape depends on which modules you have added, which no fixed test could check. Verify it by running uv run python scripts/run_pipeline.py and checking the metrics match what you already got by running the steps by hand in the notebook above.

Reflection

Files: reflection/session_3/module_2.md

Details

Question: Why does dropping high-missing columns have to happen before the per-city fill, not after? What would columns_above_missing_threshold report if you ran it on already-filled data instead? And why threshold 0.7 specifically — what would change with a lower threshold, like 0.5?