https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/index.html Presentation & objectives In this part of the course, we will discover and apply Elasticsearch, a powerful distributed search and analytics engine built on Apache Lucene. In particular, we will cover the following concepts: Understanding the ELK Stack (Elasticsearch, Logstash, Kibana) architecture and data pipeline Exploring document structure and field types in Elasticsearch Mastering Query DSL (Domain Specific Language) for searching and filtering data Understanding mapping and the difference between text and keyword field types Working with aggregations for data analytics and insights Querying arrays of objects and understanding object vs nested types By the end of these activities, you will be able to: Hugo en-us https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_1/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_1/index.html Duration30min AI Banned Introduction Before diving into Elasticsearch queries and data analysis, we need to ensure your development environment is properly configured. This section will guide you through deploying the ELK stack, verifying the infrastructure, and testing the data generator. Think of this as preparing your laboratory before the experiment—we’ll make sure all the equipment is working before we start the real work. https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_2/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_2/index.html Duration45min AI Banned Introduction In this workshop, you’ll explore a real-time e-commerce data pipeline where transactions flow from a Python generator through Logstash into Elasticsearch, ready for analysis in Kibana. Imagine you’re a data engineer at an online retail company. Your mission: understand the data infrastructure and explore the structure of incoming transactions. By the end of this session, you’ll have mastered: https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_3/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_3/index.html Duration45min AI Banned Introduction Now that you understand the Elastic Stack architecture and document structure, it’s time to master querying and filtering data. In this workshop, you’ll learn how to precisely search for documents, filter results, and understand the critical difference between text and keyword fields. By the end of this session, you’ll be able to write production-ready queries that efficiently retrieve exactly the data you need. https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_4/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_4/index.html Duration45min AI Banned Introduction In the previous workshops, you’ve explored querying and filtering data in Elasticsearch. But have you wondered how Elasticsearch knows which fields are text, which are numbers, and how it stores arrays of objects? In this workshop, you’ll dive into mapping—Elasticsearch’s schema definition. You’ll discover how Elasticsearch automatically mapped your transaction data and understand a critical limitation when querying arrays of objects. https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_5/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/practical_5/index.html Duration1h AI Banned Introduction Now that you understand field types and mapping, it’s time to master advanced query operations and data analytics. In this workshop, you’ll learn how to control query results through sorting and filtering, then analyze data at scale using aggregations. By the end of this session, you’ll be able to efficiently retrieve precisely the data you need and extract meaningful insights from your e-commerce transactions. https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/elk_documentation/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/elk_documentation/index.html This page consolidates all Elasticsearch documentation links useful for the workshop exercises. Elasticsearch Reference Query Types Search API - Main search endpoint Query DSL - Query Domain Specific Language overview Term Query - Exact matching Match Query - Full-text search Range Query - Numeric and date ranges Boolean Query - Combine multiple conditions Exists Query - Check if field exists Search Modifiers Source Filtering - Select which fields to return Sort - Order results From/Size Pagination - Paginate through results Date Math - Relative date expressions Aggregations Aggregations Overview - Introduction to aggregations Terms Aggregation - Group by field values Metrics Aggregations - Calculate statistics Understanding Query Context Query vs Filter Context - Scoring vs filtering https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/introduction/index.html Mon, 01 Jan 0001 00:00:00 +0000 https://hub.imt-atlantique.fr/data-engineering-foundations/elasticsearch/introduction/index.html Search & Analytics The ELK Stack Module 4 — Data Engineering Foundations The Problem Millions of events. How do you find the needle in the haystack? Modern systems generate data volumes that traditional databases cannot handle in real time: Application logs — thousands of lines per second Transactions — continuous stream of business events Metrics — infrastructure and performance data What you need An engine built for fast ingestion, full-text search, and on-the-fly analytics — not a relational database.