Data Engineering Foundations
Course Introduction
Contacts: Gregory Smits & Laurent Brisson
What is Data Engineering?
The practice of designing, building, and maintaining infrastructure that enables organizations to collect, store, transform, and serve data at scale.
The Mission
Connect all pieces of the data ecosystem — from raw data sources to analytics dashboards, ML models, and applications.
Data Engineering
Key Roles
| Data Engineer | Data Scientist | |
|---|---|---|
| Focus | Infrastructure & Pipelines | Models & Analysis |
| Main tasks | Build data systems, ensure data quality | Develop ML models, extract insights |
| Output | Reliable pipelines, clean datasets | Predictive models, recommendations |
| Tools | SQL, Python, Docker, dbt, Airflow | Python, R, TensorFlow, scikit-learn |
Data Engineering
Core Concepts
1. The Challenges
The 4 V’s: Volume, Velocity, Variety, Veracity
2. The Architecture
Sources → Ingestion → Storage → Transformation → Serving
3. The Paradigms
- Storage: Relational (PostgreSQL), Semantic (RDF/SPARQL), Document (Elasticsearch),
- Processing: ETL vs ELT
Data Engineering
Track Overview
| UE | Title | Focus |
|---|---|---|
| F | Data Engineering Foundations (this course) | Relational (OLTP) to most recent paradigms. Docker, PostgreSQL, Elasticsearch, dbt. Production-ready ELT pipelines. |
| G | Advanced Big Data Architecture | Hadoop, Cassandra, Kafka, Spark Streaming. Big Data cluster architectures. |
| H | Business Intelligence | OLAP, dimensional modeling, dashboard design. Strategic decision-making. |
Eco-system
Which DBMS to choose?
Problem
Which model - system - tools for my data?
A key question in the software development project, and a choice that is difficult to change later without consequences
Criteria
- nature of the data (structured, semi-structured, unstructured)
- access methods (id, joins, links, ``semantics")
- integrity constraints (ACID, BASE)
- scalability (vertical, horizontal)
- constraints from the application context (team skills, speed of dev.)
Eco-system
RDBMS
Relational DataBase Management System
- historical (70’s) but still unmissable (around 90% of corporate DB)
- safe solution (Atomicity, Consistency, Isolation, and Durability)
- stable, normalized query language (SQL)
- no data redundancy (normalization)
but …
- crucial modeling phase
- data modeling changes have strong consequences
- horizontal scalability is hard to manage
Eco-system
Graph Data
Graph DB
Data modeled as graphs when links are prioritized over values
- queries with constraints regarding graph topological structures
- data defined as triples (subject; predicate; object)
- knowledge inference from data (T-Box, A-Box)
but …
- query execution time
- inter-operability
Eco-system
NO-SQL DB
Data item = Document
Model data as key-value pairs (e.g. JSON)
- flexible to handle heterogeneous data
- easy to adjust, change, share
but …
- high redundancy
- hard integrity checks
Eco-system
Vector DB
AI-prepared data
Data embedded into (latent) numerical vectors
- same representation for heterogeneous data (img, text, node, etc.)
- fast similarity search
- horizontal scaling made easy
but …
- no explicit meaning
- costly in time and space
Eco-system
No universal solution
The relevance of the model, system and tools depends on the data and their use
The CAP theorem
No ideal solution especially in distributed environment
Course Organisation
6 Technical Modules
| Module | Topic |
|---|---|
| 1 | Technical Environment - Docker, Containerization |
| 2 | Relational DBMS - PostgreSQL Administration & Optimization |
| 3 | Semantic Web - RDF, SPARQL, OWL, Knowledge Graphs |
| 4 | Search & Analytics - Elasticsearch, ELK Stack |
| 5 | ELT & Analytics Engineering - dbt |
| 6 | Pipeline Industrialization - CI/CD, Monitoring, Documentation |
Course Organisation
Learning Approach
Format
Lectures + Hands-on Labs with a strong practical focus
Progression
Each module builds on previous skills
Capstone
Real-world case studies led by industry professionals
Tools
All technologies run in Docker for reproducible environments
Course Organisation
Evaluation
Core Competencies
- [BC02] Communicate effectively about data engineering
- [BC04] Design scalable database solutions
- [BC07] Build robust data pipelines
4 Lab Reports
Each evaluates BC02 (communication) + a technical competency
| Report | Competencies |
|---|---|
| PostgreSQL Lab | BC02 + BC04 |
| Elasticsearch Lab | BC02 + BC04 |
| dbt Lab | BC02 + BC07 |
| Pipeline Project | BC02 + BC07 |