Data Engineering Pipeline & Industrialization

Presentation & objectives

This Data Engineering course introduces students to building modern data pipelines using Azure, Python and industrial practices. It emphasizes the bronze/silver/gold approach to transform raw data into actionable insights, while integrating data quality, testing and generative AI. In particular, we will cover the following concepts:

• Understanding the Bronze/Silver/Gold architecture for layered data modeling
• Mastering ELT paradigm and modern pipeline design patterns
• Implementing clean code principles and Python structuration (classes, error handling)
• Writing unit tests and data quality tests with pytest
• Setting up CI/CD pipelines with GitHub Actions
• Implementing logging and observability (structured JSON logs)
• Configuring Azure monitoring and cost tracking
• Using generative AI tools appropriately (Copilot OK with tests)

By the end of this course, you will be able to:

• Configure an Azure/GitHub environment and implement a complete ELT pipeline (bronze → silver → gold) with logs and unit tests
• Validate data quality via pytest and data quality metrics
• Deploy a CI/CD workflow via GitHub Actions and monitor costs/performance

Activities

Day 1: ELT Foundations - Setup + Design

1. Introduction to Data Engineering: Understand the role of a Data Engineer, compare ELT vs ETL paradigms, and discover the Bronze/Silver/Gold architecture
2. Data Modeling and Environment Setup: Profile bronze-level data, define silver/gold specifications, create a pipeline diagram, and configure Azure SQL DB with GitHub repository
3. Technology Stack and Scalability: Explore technology choices (Azure SQL, sqlalchemy, pandas) and learn scalable design patterns (SQL MERGE/upsert, partitioning)
4. ELT Pipeline Implementation: Build a complete pipeline with extract(), transform(), and load() functions from bronze to silver to gold layers
5. Data Quality and Testing Introduction: Learn to read technical documentation, implement pytest for data quality, and discuss appropriate AI tool usage

Homework: Complete the ELT pipeline, write unit and integration tests, implement structured JSON logs, and create silver→gold aggregates.

Day 2: Industrialization + Evaluation

1. Code Quality Review: Master advanced pytest techniques, implement logging observability, and handle SQL errors properly
2. Testing Integration: Integrate tests and logs into the complete bronze→gold pipeline using SQL mocks for unit testing
3. CI/CD and Monitoring: Set up GitHub Actions workflows for automated testing, building, and Azure deployment, with introduction to Prefect orchestration
4. Evaluation Project: Configure monitoring dashboards, track Azure costs, and package the project for production
5. Final Presentation: Prepare and deliver a 5-minute video presentation of your work

Assessment

Deliverables 60% and 5-min Video Presentation per group 40%