https://hub.imt-atlantique.fr/data-engineering-foundations/index.html Presentation & objectives This course covers fundamental concepts and practical skills required in data engineering, from traditional relational databases to modern paradigms. It emphasizes hands-on experience with essential tools and frameworks, preparing students to design, implement, and maintain robust data infrastructure in real-world scenarios. The course also incorporates best practices in data governance, security, and environmental impact considerations. The application project enables students to design and automate a complete ELT pipeline including data transformation, data quality validation tests, deployment of a vector database to feed a RAG (Retrieval-Augmented Generation) system that enhances LLM capabilities with domain-specific knowledge, and implementation of a CI/CD pipeline with comprehensive instrumentation. Hugo en-us Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/tu_introduction/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/tu_introduction/index.html Data Engineering Foundations Course Introduction Contacts: Gregory Smits & Laurent Brisson What is Data Engineering? The practice of designing, building, and maintaining infrastructure that enables organizations to collect, store, transform, and serve data at scale. https://hub.imt-atlantique.fr/data-engineering-foundations/technical_environment/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/technical_environment/index.html Presentation & objectives In a database course, every student needs to run services such as PostgreSQL, MySQL, Redis, ElasticSearch or MongoDB on their own machine. But students use a wide variety of systems—Windows, macOS, Linux, different versions, different configurations—which makes installing and configuring databases a time-consuming and error-prone task. Docker solves this problem by providing isolated, reproducible environments that behave the same on every computer. Instead of struggling with installers, system dependencies, port conflicts, or path issues, students can launch a fully configured database in seconds with a single command. This ensures that everyone works with identical setups, removes “it works on my machine” issues, and lets us focus on learning database concepts rather than debugging local environments. https://hub.imt-atlantique.fr/data-engineering-foundations/relational_dbms/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/relational_dbms/index.html Presentation & objectives This part of the track is dedicated to a major data model, relational data. Even if it is not the most appropriate choice for very large datasets, the relational data model is still predominant to store the data of most dynamic web-based applications. Two topics are addressed and separated into two practical activities. The first one is dedicated to database servers administration. A focus is given to the Postgresql Relational DataBase Management System (RDBMS). More precisely, by the end of this first activity, you will be able to: https://hub.imt-atlantique.fr/data-engineering-foundations/semantic_web/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/semantic_web/index.html Presentation & objectives In this part of the course, we will discover and apply the standards of the Semantic Web and the Web of Data (languages and principles). In particular, we will cover the following concepts: The Semantic Web and linked data through the example of the Google search engine and Wikidata. Representation of linked data (RDF model) Querying linked data using the SPARQL language Expression of an ontology using the OWL language and Protégé software Reasoning and deduction of knowledge using the SWRL language By the end of these activities, you will be able to: https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/index.html Presentation & objectives In this part of the course, we will discover and apply Elasticsearch, a powerful distributed search and analytics engine built on Apache Lucene. In particular, we will cover the following concepts: Understanding the ELK Stack (Elasticsearch, Logstash, Kibana) architecture and data pipeline Exploring document structure and field types in Elasticsearch Mastering Query DSL (Domain Specific Language) for searching and filtering data Understanding mapping and the difference between text and keyword field types Working with aggregations for data analytics and insights Querying arrays of objects and understanding object vs nested types By the end of these activities, you will be able to: https://hub.imt-atlantique.fr/data-engineering-foundations/elt_dbt/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elt_dbt/index.html Presentation & objectives This course offers a complete journey to understand modern data architectures and put them into practice. We will clarify the difference between ETL vs ELT and specify the roles of a data lake, data warehouse, and lakehouse — principles, use cases, and trade-offs. An ELT project with dbt will illustrate data processing strategies, dimension historization, and the basics of data quality (testing, documentation, control). By the end of this course, students will be able to: https://hub.imt-atlantique.fr/data-engineering-foundations/pipeline/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/pipeline/index.html Presentation & objectives This Data Engineering course introduces students to building modern data pipelines using Azure, Python and industrial practices. It emphasizes the bronze/silver/gold approach to transform raw data into actionable insights, while integrating data quality, testing and generative AI. In particular, we will cover the following concepts: Understanding the Bronze/Silver/Gold architecture for layered data modeling Mastering ELT paradigm and modern pipeline design patterns Implementing clean code principles and Python structuration (classes, error handling) Writing unit tests and data quality tests with pytest Setting up CI/CD pipelines with GitHub Actions Implementing logging and observability (structured JSON logs) Configuring Azure monitoring and cost tracking Using generative AI tools appropriately (Copilot OK with tests) By the end of this course, you will be able to: https://hub.imt-atlantique.fr/data-engineering-foundations/assessment/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/assessment/index.html Overview Your final assessment consists of two complementary mini-projects that demonstrate your mastery of machine learning engineering principles across different domains. Both projects emphasize the transition from exploratory data science to production-ready ML systems, showcasing your ability to build structured, maintainable, and well-documented machine learning solutions. Core competencies Your work will be assessed based on three fundamental competencies that align with the official program objectives: [BC02] Communicate effectively about data engineering concepts and solutions [BC04] Design and implement scalable database solutions using both relational and other paradigms [BC07] Create and maintain robust data pipelines for ETL processes while applying best practices in data governance, security, and environmental consideration Read carefully the expectations for these different competencies. https://hub.imt-atlantique.fr/data-engineering-foundations/integrity/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/integrity/index.html Team Collaboration Policy Projects are pair work between two students. Discussion of concepts and approaches is encouraged within teams and across the class. Code sharing between different teams results in automatic failure for all involved parties. All external sources must be properly cited. AI Tool Usage Guidelines AI Banned AI Allowed AI usage depends on activity badges. Using AI during AI Banned activities results in automatic course failure. For AI Allowed activities, teams must maintain records of AI interactions and demonstrate genuine understanding of AI-generated code. Documentation Requirements: Each project README must include a section describing AI tool usage, specifying which tools were used and for what purposes. Teams not using AI must explicitly state this. Failure to document AI usage properly may result in academic integrity violations. https://hub.imt-atlantique.fr/data-engineering-foundations/corrections/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/corrections/index.html This section is reserved for exercise corrections.