5. Hyperparameter Tuning

Duration1h
Learning Objectives

Understand:
[BC07] why a more powerful model can underperform a simpler one until it is properly configured

Apply:
[BC07] a systematic search for a model’s best configuration, instead of guessing it by hand
[BC04] test-driven development to build a reusable configuration-search tool

Prerequisites

Introduction

Module 4 left you with 8 RFE-selected columns and a LinearRegression baseline around RMSE 27.6. This module asks whether a more flexible model — XGBRegressor — can do better, and what happens if you use it without tuning it at all. uv sync --group ml is required for this module (xgboost).

Notebook

Files: notebooks/session3/05_hyperparameter_tuning.ipynb

Open it and select the same .venv kernel as your other notebooks. Work through the cells in order:

  1. Rebuild the Module 4 pipeline: cleaned dataset, then the 8 RFE-selected columns
  2. Evaluate a plain, untuned XGBRegressor on those same 8 columns with evaluate_group_cv — compare to Module 4’s LinearRegression
  3. Once tune_xgboost is complete (see Implementation below), apply it with a small grid ({"max_depth": [3, 5], "learning_rate": [0.05, 0.2]}, 4 combinations) and look at every combination tried, not just the best one

Implementation

Files: src/air_quality/tuning.py, src/air_quality/workflows.py

  • Complete tune_xgboost in the new src/air_quality/tuning.py (already scaffolded) — its docstring and tests/test_tuning.py specify exactly what it should do
  • Update run_advanced in src/air_quality/workflows.py once more: replace the model with XGBRegressor configured with the best parameters found. Verify with uv run python scripts/run_pipeline.py, never from the notebook.
GridSearchCV + GroupKFold: the tool version of Module 3

GridSearchCV(..., cv=GroupKFold(n_splits=...)) gives you the exact same guarantee you built by hand in Module 3 — a group never appears in both train and validation for a given fold — applied automatically across every hyperparameter combination instead of manually across every city.

Why rmse, mae and r2 together

tune_xgboost scores every combination on all three metrics regression_metrics() has always reported — not just rmse. A search still needs one number to rank combinations against (here, rmse, via refit="rmse"), but there is no reason to throw the other two away once they are computed for free; you can read all three for whichever combination wins.

Reflection

Files: reflection/session_3/module_5.md

Details

Question: Compare four numbers: the untuned XGBRegressor, the worst combination in the grid, the best combination, and Module 4’s LinearRegression on the same 8 columns. Was it worth introducing a more complex model here? What does the gap between the best and worst combination tell you about why tuning matters, independently of which model you picked?