Data Engineering Foundations

Presentation & objectives

This course covers fundamental concepts and practical skills required in data engineering, from traditional relational databases to modern paradigms. It emphasizes hands-on experience with essential tools and frameworks, preparing students to design, implement, and maintain robust data infrastructure in real-world scenarios. The course also incorporates best practices in data governance, security, and environmental impact considerations. The application project enables students to design and automate a complete ELT pipeline including data transformation, data quality validation tests, deployment of a vector database to feed a RAG (Retrieval-Augmented Generation) system that enhances LLM capabilities with domain-specific knowledge, and implementation of a CI/CD pipeline with comprehensive instrumentation.

Key concepts covered:

  • Advanced relational database concepts (indexing, security, rights management, limitations, distributed relational models)
  • Data paradigms (document, semantic)
  • Data querying tools and techniques
  • Data pipelines: automation, performance, quality, and validation (ETL)
  • Data governance, rights management, security, and environmental impact

Prerequisites :

  • Proficiency in SQL and database management
  • Familiarity with Linux command-line interface
  • Programming experience (preferably in Python or Java)

Assessment

By the end of this course, students will be able to:

  • [BC-02] Communicate effectively about data engineering concepts and solutions
  • [BC-04] Design and implement scalable database solutions using both relational and other paradigms
  • [BC-07] Create and maintain robust data pipelines for ETL processes while applying best practices in data governance, security, and environmental considerations

The evaluation will be based on case studies and a knowledge quizz.

License

This content is made available under the Creative Commons Attribution-NonCommercial 4.0 International License.

Citation

To cite this educational material, please use the following reference:

Laurent Brisson, Fahima Djelil, Grégory Smits. "Data Engineering Foundations". IMT Atlantique. Available under CC BY-NC 4.0 license. https://hub.imt-atlantique.fr/data-engineering-foundations/

You are free to:

  • Share — copy and redistribute the material in any medium or format
  • Adapt — remix, transform, and build upon the material

Under the following terms:

  • Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made
  • NonCommercial — You may not use the material for commercial purposes