3. Robust Evaluation

Duration1h
Learning Objectives

Apply:
[BC07] a one-off evaluation generalized into a systematic, repeatable check across every relevant group in the data
[BC04] test-driven development to build a reusable, trustworthy evaluation tool

Analyze:
[BC07] why a single train/test result is not enough evidence to trust a model, and what a more systematic evaluation adds

Prerequisites

Introduction

Session 2’s reflection question asked you to compare training on Kampala and testing on Nairobi, then reversing it — two different, both disappointing results, and neither one told you anything about Lagos or Bujumbura. A single manual train/test pair only ever measures one direction between two cities. This module generalizes it: GroupKFold evaluates against every city in turn, so no city is ever left unexamined.

Notebook

Files: notebooks/session3/03_group_kfold_evaluation.ipynb

Open it and select the same .venv kernel as your other notebooks. Work through the cells in order:

  1. Rebuild the cleaned, 4-city dataset, reusing the Module 2 functions you already completed
  2. See why one manual pair cannot tell you how the model performs on the cities it never involved
  3. Once evaluate_group_cv is complete (see Implementation below), apply it across all four cities and read the per-city breakdown, then the aggregated mean/std

Implementation

Files: src/air_quality/evaluation.py, src/air_quality/workflows.py

  • Complete evaluate_group_cv in src/air_quality/evaluation.py (already scaffolded, in the “Session 3 — optional modules” section) — its docstring and tests/test_evaluation_advanced.py specify exactly what it should do
  • Update run_advanced in src/air_quality/workflows.py — the one you wrote in Advanced Cleaning: keep the same cleaning (scope, drop, fill, add temporal features), but replace the manual-split evaluation at the end with evaluate_group_cv across every city. config.train_city/config.test_city are no longer used at this point; remove them from AdvancedPipelineConfig if nothing else in your code still needs them. There is deliberately no fixed test for run_advanced’s exact shape — verify it with uv run python scripts/run_pipeline.py (never from the notebook), comparing its result to the per-city breakdown and aggregate you already saw there.
A fresh model per fold

Each fold must fit its own model, never the same instance refit repeatedly across folds — otherwise a fold could carry over something learned from a previous one. evaluate_group_cv’s docstring points you to sklearn.base.clone() for this.

Reflection

Files: reflection/session_3/module_3.md

Details

Question: Which city has the highest RMSE when held out? Does the mean RMSE across all four folds match either of Session 2’s two numbers? What does a large std across folds tell you about how much you should trust a performance number computed from a single train/test pair, the way Session 2 did?