5. Evaluation & Business Impact

Duration2h30 AI Allowed

Introduction

This final practical session focuses on the Evaluation phase of CRISP-DM, where you synthesize your technical achievements into meaningful business insights. You’ll leverage all the tools developed in previous sessions to conduct a comprehensive analysis and complete your professional README for the final assessment.

CRISP-DM Process CRISP-DM Process

Source: IBM CRISP-DM Help Overview

The CRISP-DM Evaluation phase is crucial for determining whether your models meet business objectives and are ready for deployment. This session guides you through systematic model comparison, performance interpretation, and business impact assessment.

Learning Objectives

By the end of this session, you will be able to:

  • Conduct systematic model comparison using the tools you’ve developed
  • Interpret ML performance metrics in the context of real-world business constraints
  • Analyze geographic and temporal patterns in model performance
  • Formulate actionable business recommendations based on quantitative analysis
  • Document professional insights suitable for stakeholder communication
  • Complete a comprehensive README that demonstrates your analytical capabilities

Prerequisites

This session assumes you have completed all previous workshops and have:

  • A fully functional ML pipeline with multiple models (Linear, XGBoost, LightGBM)
  • MLflow experiment tracking capabilities
  • Hyperparameter optimization functionality
  • Understanding of cross-validation with geographic grouping

Problem Statement

You have successfully developed a complete machine learning pipeline for air quality prediction across four African cities. Now comes the critical question: How do you interpret your technical results in the business context?

Your mission is to transform your technical achievements into actionable business insights by analyzing model performance against real-world constraints. You’ll examine why certain cities are more challenging to predict, what this means for public health monitoring, and how your findings should influence deployment decisions.

Your final deliverable is a comprehensive README that documents the complete case study, including:

  1. Context & Business Understanding: Present the air quality prediction problem, the four African cities studied, and the public health implications
  2. Objectives & Methodology: Describe your analytical approach, including:
    • Data processing and feature engineering strategies
    • Model selection rationale (Linear Regression, XGBoost, LightGBM)
    • Cross-validation strategy with geographic grouping
    • Hyperparameter tuning methodology with detailed explanation of your optimization approach
  3. MLOps Implementation: Document your technical workflow using:
    • MLflow screenshots showing experiment tracking, parameter logging, and model comparison
    • Evidence of hyperparameter optimization runs with performance evolution
    • Model registry and versioning approach
  4. Results & Interpretation: Analyze model performance using:
    • RMSE contextualized against WHO air quality guidelines
    • Geographic and temporal performance patterns
    • Comparison across different model types and hyperparameter configurations
  5. Business Recommendations: Provide actionable insights for stakeholders on deployment decisions and model selection

Rather than simply reporting technical metrics, your README should tell the complete story of your data science project, demonstrating both your analytical capabilities and MLOps best practices.

Assessment

Your project will be evaluated on the following criteria:

Technical Implementation (60%)

  • ✅ Complete implementation of DataProcessor, FeatureEngineer, ModelTrainer, and Evaluator classes
  • ✅ Geographic cross-validation using GroupKFold to prevent data leakage
  • ✅ Hyperparameter optimization with XGBoost and LightGBM models
  • ✅ MLflow integration across all pipeline components
  • ✅ Comprehensive performance analysis across different cities and model types

MLOps & Documentation (40%)

  • ✅ README completeness: Full case study documentation including context, objectives, methodology, results, and recommendations
  • ✅ Hyperparameter tuning documentation: Clear explanation of optimization strategy with evidence from MLflow experiments
  • ✅ MLflow evidence: Screenshots demonstrating:
    • Experiment tracking with multiple runs
    • Parameter and metric logging
    • Hyperparameter optimization results and performance comparison
    • Model versioning and selection process
  • ✅ Reproducibility: Clear instructions and documentation allowing others to understand and reproduce your work
  • ✅ Business interpretation: Meaningful analysis connecting technical results to public health implications

Submission process

  1. Commit all pending files in your repository.
  2. In the project folder root, run uv run python scripts/run_tests.py.
  3. Add and commit the two new files test_proof.json and test_results.json without modifying them.
  4. Publish on moodle the link toward your repository (https version). Please check your repository allow public access.

Failure to submit your project will result in failing the course.