3. Hyperparameter Optimization

Duration2h30 AI Banned

Introduction

This practical session extends your structured ML pipeline by introducing advanced algorithms and systematic hyperparameter optimization. You’ll transform your linear regression baseline into a comprehensive comparison framework supporting gradient boosting models with automated parameter tuning.

Through this hands-on workshop, you will master advanced machine learning engineering skills:

• Advanced Algorithms: Implement XGBoost and LightGBM for improved predictive performance
• Hyperparameter Optimization: Apply GridSearchCV with geographic cross-validation to prevent data leakage
• Model Comparison: Systematically evaluate multiple algorithms with statistical rigor
• Production Optimization: Learn when and how to apply optimization in real-world scenarios
• Performance Analysis: Understand the trade-offs between model complexity, training time, and predictive accuracy

The challenge is to extend your existing pipeline architecture while maintaining geographic cross-validation integrity and implementing robust optimization strategies. This represents the natural evolution from proof-of-concept to production-ready ML systems.

Prerequisites

Before starting this activity, ensure you have completed:

• Environment Setup - Python environment with required packages installed
• Structured ML Pipeline Development - Previous workshop with complete pipeline implementation

This activity assumes you have a working structured pipeline with DataProcessor, FeatureEngineer, ModelTrainer, and Evaluator components fully implemented.

Understanding Advanced Model Integration

The advanced pipeline extends your existing architecture while preserving the modular design principles:

src/
├── pipeline/
│   ├── data_processor.py      # ✅ Already implemented
│   ├── feature_engineer.py    # ✅ Already implemented  
│   ├── model_trainer.py       # 🔧 Extend with XGBoost/LightGBM
│   └── evaluator.py           # 🔧 Add hyperparameter optimization
└── utils/
    └── config.py              # 🔧 Add model configurations

New Components

• XGBoost Integration: Gradient boosting framework optimized for speed and performance
• LightGBM Integration: Microsoft’s efficient gradient boosting framework
• GridSearchCV with Geographic Folds: Systematic hyperparameter search that respects city boundaries
• Performance Comparison: Automated evaluation across multiple algorithms and configurations

Architecture Benefits

• Backward Compatibility: Existing linear models continue to work unchanged
• Extensible Design: Easy addition of new algorithms without breaking existing code
• Geographic Integrity: Hyperparameter optimization respects data leakage constraints
• Production Ready: Configurable optimization suitable for deployment scenarios

Getting Started

Environment Update

First, ensure you have the required advanced ML libraries xgboost 3.0.2 and lightgbm 3.3.0. Add them to a ml group with uv.

1
2
3
4
5

# Update your environment with new dependencies using uv

# Verify installations
uv run python -c "import xgboost; print(f'XGBoost: {xgboost.__version__}')"
uv run python -c "import lightgbm; print(f'LightGBM: {lightgbm.__version__}')"

Baseline Performance Assessment

Before implementing advanced models, establish your current baseline:

1
2
3
4

# Test current linear model performance
uv run python scripts/run_pipeline.py --model linear --verbose

# This should complete successfully with your previous implementation

Implementation Guide

You will extend your existing pipeline by implementing support for advanced algorithms and optimization. Complete the following components in order:

Phase 1: Model Configuration Extension

First, update the configuration to support new model types and optimization parameters.

Configuration Updates

Update src/utils/config.py to include new model types and hyperparameter grids.

Reference Documentation: Scikit-learn Model Selection

Sections to Complete:

• Extend MODEL_TYPES list to include 'xgboost' and 'lightgbm'
• Add DEFAULT_PARAM_GRIDS dictionary with optimization parameters

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# TODO Add XGBoost and LightGBM support (Workshop 3)
# Available model types
MODEL_TYPES = ["linear", "xgboost", "lightgbm"]

# Default hyperparameter grids for optimization
DEFAULT_PARAM_GRIDS = {
    "xgboost": {
        'n_estimators': [100, 200],
        'max_depth': [3, 5, 7],
        'learning_rate': [0.01, 0.1, 0.2],
        'subsample': [0.8, 1.0],
        'colsample_bytree': [0.8, 1.0]
    },
    "lightgbm": {
        'n_estimators': [100, 200],
        'max_depth': [3, 5, 7],
        'learning_rate': [0.01, 0.1, 0.2],
        'subsample': [0.8, 1.0],
        'colsample_bytree': [0.8, 1.0]
    }
}
# TODO_END

Phase 2: ModelTrainer Extension

Extend the ModelTrainer class to support advanced algorithms while maintaining the existing interface.

Method Extension: create_model()

Purpose: Extend model factory to support XGBoost and LightGBM instances.

Reference Documentation:

• XGBoost Python API
• LightGBM Python API

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

def create_model(self, model_type, **params):
    """
    Create a model instance of the specified type.
    
    Factory method that instantiates different types of regression models
    with appropriate default configurations and custom parameters.
    
    Parameters
    ----------
    model_type : str
        Type of model ('linear', 'xgboost', 'lightgbm')
    **params : dict
        Model-specific parameters to override defaults
        
    Returns
    -------
    sklearn-compatible estimator
        Initialized model instance ready for training
        
    Raises
    ------
    ValueError
        If model_type is not supported
    ImportError
        If required libraries are not installed
    """

Sections to Complete:

• Add import statements for XGBoost and LightGBM
• Extend the model creation logic with new cases for 'xgboost' and 'lightgbm'
• Configure appropriate default parameters (objective, random_state, etc.)

Phase 3: Hyperparameter Optimization

Implement systematic hyperparameter optimization that respects geographic cross-validation constraints.

Method: hyperparameter_optimization_cv()

Purpose: Perform GridSearchCV with geographic cross-validation to find optimal hyperparameters while preventing data leakage.

Reference Documentation:

• GridSearchCV
• GroupKFold
• Cross-validation with grouped data

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

def hyperparameter_optimization_cv(self, model, param_grid, X, y, groups=None):
    """
    Perform hyperparameter optimization using GridSearchCV with geographic cross-validation.
    
    This method combines GridSearchCV with GroupKFold to ensure that entire cities
    are either in training OR validation during hyperparameter search, preventing data leakage.
    
    Parameters
    ----------
    model : sklearn estimator
        Scikit-learn model to optimize
    param_grid : dict
        Dictionary of hyperparameters to search over
    X : pandas.DataFrame
        Feature matrix
    y : pandas.Series  
        Target variable
    groups : pandas.Series, optional
        Grouping variable for GroupKFold (e.g., cities)
        
    Returns
    -------
    tuple
        Tuple containing:
        - best_model : sklearn estimator
            Best model found during search
        - best_params : dict
            Best hyperparameters found
        - best_score : float
            Best RMSE score achieved
            
    Notes
    -----
    Uses 'neg_root_mean_squared_error' scoring which returns negative RMSE values.
    The method converts these back to positive values for interpretability.
    """

Key Implementation Details:

1. Geographic Cross-Validation Setup: Use GroupKFold when groups are provided to respect city boundaries
2. Scoring Configuration: Use 'neg_root_mean_squared_error' for consistent RMSE optimization
3. Parallel Processing: Set n_jobs=-1 for faster optimization
4. Verbose Output: Control verbosity based on logging level

Phase 4: Pipeline Integration

Integrate optimization functionality into the main pipeline execution with user-controllable options.

Script Extension: run_pipeline.py

Purpose: Integrate hyperparameter optimization into the main pipeline execution using the existing TODO structure.

Sections to Complete:

The run_pipeline.py script already contains properly placed TODO sections for Workshop 3. You need to complete the existing TODO blocks that handle:

1. Import Parameter Grids: Complete the existing import TODO for DEFAULT_PARAM_GRIDS
2. Pipeline Logic: Implement the conditional optimization logic in Step 4’s existing TODO blocks

Pipeline Logic Implementation:

The optimization logic should follow this pattern in the existing TODO structure:

• No Optimization: Use standard cross-validation with default parameters
• Optimization Requested: Apply GridSearchCV with geographic folds, then evaluate optimized model
• No Parameter Grid: Fall back to standard cross-validation with warning message

Key Implementation Points:

• Use DEFAULT_PARAM_GRIDS.get(args.model, {}) to retrieve parameter grids
• Check if parameter grid is empty and handle gracefully
• Apply the hyperparameter_optimization_cv() method you implemented in the Evaluator
• Ensure fallback to standard cross-validation when optimization fails

Testing and Validation

Progressive Testing Strategy

Test each implementation phase incrementally to ensure stability:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Phase 1: Test new model creation
uv run python scripts/run_pipeline.py --model xgboost
uv run python scripts/run_pipeline.py --model lightgbm

# Phase 2: Test optimization functionality  
uv run python scripts/run_pipeline.py --model xgboost --optimize
uv run python scripts/run_pipeline.py --model lightgbm --optimize

# Phase 3: Test with different configurations
uv run python scripts/run_pipeline.py --model xgboost --optimize --n-features 20
uv run python scripts/run_pipeline.py --model lightgbm --optimize --method rfe

Advanced Topics

Understanding Hyperparameter Grids

The default parameter grids are designed to balance exploration with computational efficiency:

Coarse-to-Fine Strategy:

• Initial Grid: Covers wide parameter ranges with few values
• Refined Grid: Can be narrowed based on initial results for production use

Key Parameters Explained:

• n_estimators: More trees generally improve performance but increase training time
• max_depth: Controls overfitting; deeper trees capture more complex patterns
• learning_rate: Lower values require more estimators but often achieve better performance
• subsample/colsample_bytree: Regularization parameters that prevent overfitting

Geographic Cross-Validation Benefits

Your implementation maintains geographic integrity during optimization:

1
2
3
4
5

# Without geographic awareness (problematic)
GridSearchCV(model, params, cv=5)  # May split cities across folds

# With geographic awareness (correct)
GridSearchCV(model, params, cv=GroupKFold(n_splits=4))  # Keeps cities together

Why This Matters:

• Data Leakage Prevention: Ensures model cannot memorize city-specific patterns
• Realistic Performance: Evaluates generalization to completely new locations
• Deployment Validity: Mirrors real-world scenario of predicting for new cities

Optimization Trade-offs

Understanding when to use optimization in production:

When to Optimize:

• Initial model development and research phases
• Periodic retraining with new data
• Model performance is critical for business outcomes

When to Skip Optimization:

• Real-time prediction requirements
• Computational resources are limited
• Model performance is already satisfactory

Slow Optimization Performance

To speed up hyperparameter search:

1
2
3
4
5
6
7
8

# Add early stopping for tree-based models
GridSearchCV(
    estimator=model,
    param_grid=param_grid,
    cv=cv_strategy,
    n_jobs=-1,          # Use all CPU cores
    verbose=1           # Monitor progress
)

Conclusion

This workshop has extended your ML pipeline with advanced algorithms and systematic optimization capabilities. By completing this implementation, you have gained practical experience with:

• Advanced Algorithms: Integrating gradient boosting models (XGBoost, LightGBM) into existing pipelines
• Hyperparameter Optimization: Implementing GridSearchCV with geographic cross-validation constraints
• Production Considerations: Balancing model performance improvements with computational costs
• Architecture Extension: Maintaining backward compatibility while adding new functionality

The optimization techniques you’ve learned represent industry best practices for systematic model improvement. This approach ensures that performance gains are statistically valid and will generalize to new data.