6. Experiment Tracking

Introduction

Imagine you are the one accountable for the air quality model a city actually deploys. Over the past few weeks you tried a dozen configurations — different cleaning thresholds, different feature subsets, different hyperparameters — and one of them clearly performed best. Now someone asks: which exact configuration was that, what did it see, and can you reproduce it? If the honest answer is “let me scroll back through my terminal history and hope I remember,” that is not a coding problem — it is a business one. A model nobody can reliably reproduce, justify, or hand off is not one an organization can trust in production.

That is the moment tracking earns its place: not while you are still exploring one or two ideas by hand, but once the number of things you have tried outgrows what a person can keep in their head or a terminal scrollback can hold. MLflow’s answer is to log every run — its configuration and its result — somewhere queryable, so “which configuration won, and why” becomes a query instead of an archaeology exercise.

Core vocabulary

MLflow has four components: Tracking (log parameters, metrics, and files for each run), Projects (package code so it runs the same way anywhere), Models (a standard format for saving a trained model), and Model Registry (a central store for promoting a model from “just trained” to “in production”). This module only uses Tracking — the other three matter more at team/production scale, and come back in the “Going further” section below.

Inside Tracking, three words matter:

• Experiment — a named group of related runs (you will use one experiment, "air_quality", for everything).
• Run — one execution: one call to your pipeline, with its own parameters and metrics.
• Parameters vs. Metrics — MLflow keeps these strictly separate. Parameters are the inputs you chose (missing_threshold=0.7, arbitrary types); metrics are the numeric outputs you measured (rmse_mean=27.58, numbers only). Passing a metric where a parameter is expected (or vice versa) is a common, confusing error — keep the two dicts you build clearly apart.

Implementation

Files: src/air_quality/tracking.py, scripts/run_tracked_tuning.py

1. Complete log_run in the new src/air_quality/tracking.py (already scaffolded) — its docstring and tests/test_tracking.py specify exactly what it should do. Run uv run pytest tests/test_tracking.py -v until it passes. uv sync --group mlops is required first.
2. Complete the TODO in the new scripts/run_tracked_tuning.py (also already scaffolded). It rebuilds Module 4’s cleaned, RFE-selected dataset and reruns Module 5’s tune_xgboost grid search for you — it stops right before logging. Your job: loop over tuning_result["cv_results"] (one {"params": ..., "rmse": ..., "mae": ..., "r2": ...} entry per grid point already tried) and log each one as its own run with tracking.log_run. Then run it:

1

uv run python scripts/run_tracked_tuning.py

This logs all 4 combinations from Module 5’s grid in one go — a far more interesting comparison than any single run on its own.

3. Launch the UI from the project root and keep the terminal open:

1

uv run mlflow ui

Open the printed local address (typically http://127.0.0.1:5000) in your browser.

Reading the MLflow UI

You are looking at exactly the same kind of view as the screenshot below (from a much larger set of runs than the 4 you just logged — this one compares linear regression, XGBoost and LightGBM across several feature-selection methods and counts):

A few things to try, matching what is visible above:

• The experiment name (air_quality) on the left groups all your runs together.
• The Runs table shows one row per run: your run_name, when it ran, and — as their own columns — every parameter and metric you logged (max_depth, learning_rate, rmse, mae, r2, and further columns to the right).
• Click Sort on rmse to instantly find your best combination — no scrolling through terminal history or rereading Module 5’s printed table.
• The search bar accepts real queries, e.g. metrics.rmse < 28, useful once you have more than a handful of runs.
• Selecting several runs and using Compare gives you a side-by-side table and parallel-coordinates plot — worth trying once you have logged all 4 of your runs.

Reflection

Files: reflection/session_3/module_6.md

Going further: from a laptop to a team

This module writes to a local ./mlruns folder — nothing to install beyond mlflow itself, nothing to configure. That is enough for one pair of students comparing a handful of runs on one laptop. It is not what a real team uses, and it is worth knowing what changes:

• A remote tracking server (mlflow server) instead of a local folder, so everyone on the team logs to — and can query — the same place, not each other’s separate ./mlruns directories.
• A real backend store (Postgres, MySQL) instead of local files, so many people can log and query runs concurrently without corrupting anything.
• Shared or cloud artifact storage (S3, GCS, Azure Blob) instead of local disk for the actual files a run produces (trained model, plots) — a teammate, or a production server, cannot reach a file that only exists on your laptop.
• The Model Registry (the 4th component from the vocabulary above, not used in this exercise) to formally version a model and move it through stages — from “just trained” to “staging” to “production” — with an audit trail of who promoted what and when.

None of this changes what you just did conceptually — you would still call log_run the same way — only where the tracking server and the artifacts live. The model that eventually gets registered for production is also, in a real setting, the one retrained on the full dataset once you trust its evaluation — the same idea the future consolidation module comes back to.