2. Architecture & Document Structure

Duration45min AI Banned

Introduction

In this workshop, you’ll explore a real-time e-commerce data pipeline where transactions flow from a Python generator through Logstash into Elasticsearch, ready for analysis in Kibana.

Imagine you’re a data engineer at an online retail company. Your mission: understand the data infrastructure and explore the structure of incoming transactions.

By the end of this session, you’ll have mastered:

• Navigating the Elastic Stack architecture
• Understanding how data flows through the pipeline
• Analyzing document structure and field types

Learning Objectives

By completing this workshop, you will be able to:

• ✅ Explore Elasticsearch clusters using both Kibana GUI and command-line APIs
• ✅ Understand the Logstash pipeline and data transformation process
• ✅ Analyze document structure and field organization
• ✅ Identify different data types (objects, arrays, nested structures)

Prerequisites

Section 1: Architecture Discovery

Before diving into data analysis, you need to understand the infrastructure powering your e-commerce platform. In this section, you’ll explore how Elasticsearch, Logstash, and Kibana work together to ingest, store, and visualize data.

Exercise 1.1: Explore the Elastic Stack via Kibana GUI

Kibana provides a web interface for interacting with Elasticsearch. Let’s explore the available tools.

1
2

cd elasticsearch-workshop/generators/
uv run python generator.py

1. Open Kibana in your browser: http://localhost:5601

2. Navigate to Management → Stack Management → Index Management

   • Observe the list of indices
   • Note the document counts and storage size
   • Identify the index receiving transaction data

3. Open Dev Tools (left sidebar)

   • This is where you’ll execute Elasticsearch queries
   • The console shows two panels: left (your query), right (response)

4. Try this sample query to see all indices:

   1	GET /_cat/indices?v

   Click the ▶️ Play button or press Ctrl+Enter to execute.

1

curl -X GET "http://localhost:9200/_cat/indices?v"

Exercise 1.2: Explore via Command Line

While Kibana is convenient, command-line access is essential for automation and troubleshooting. Elasticsearch provides Cat APIs for human-readable output.

Use the command line to explore the cluster. All Elasticsearch APIs are accessible via HTTP requests.

Useful Documentation:

• Cluster health:
  • Cat Health API Documentation
  • Get the cluster health
• List indices: Cat Indices API Documentation
• Index statistics: Cat Count API Documentation
• Elasticsearch shards and replicas: A practical guide

1

curl http://localhost:9200/_cat/health?v

Exercise 1.3: Understand the Logstash Pipeline

Logstash processes incoming data before sending it to Elasticsearch. Understanding this pipeline is crucial for troubleshooting data issues.

1. Understand the data generator output:

If you have forgot to stop the generator stop it now. Now, let’s examine what data the generator produces by reading its help documentation:

1
2

cd elasticsearch-workshop/generators/
uv run python generator.py --help

Read the DATA OUTPUT section carefully. You’ll see examples of both transaction and application logs. Write down one example of a transaction document and one example of an application log entry.

2. Locate the Logstash pipeline configuration:

   • Check the /logstash/pipeline/ directory in the workshop repository
   • Find the configuration files for both data streams:
     • 01-transactions.conf (handles JSON transactions)
     • 02-applogs.conf (handles plain text logs)

3. Analyze the pipeline stages for BOTH configurations:

   For 01-transactions.conf:

   • Input: What port? What codec?
   • Filter: What transformations occur?
   • Output: What index does it write to?

   For 02-applogs.conf:

   • Input: What port? What codec? (different from transactions!)
   • Filter: What does the GROK pattern do? What about the KV filter?
   • Output: What index pattern does it use?

Documentation: Logstash Configuration

Section 2: Document Structure Analysis

Now that you understand the architecture and data flow, let’s explore the actual structure of documents stored in Elasticsearch. This understanding is crucial before learning to query and analyze the data.

Understanding Elasticsearch Queries

Before diving into exercises, let’s understand the basic structure of an Elasticsearch query:

1
2
3
4
5
6
7

GET /index-pattern/_search
{
  "size": 10,                           // Number of documents to return (default: 10)
  "query": {
    "match_all": {}                     // Simple query to match all documents
  }
}

We’ll explore more complex queries in the next workshop. For now, we’ll use simple queries to retrieve and examine documents.

Exercise 2.1: Explore Document Response

Let’s start by retrieving a simple document to understand its structure and the response format.

Exercise 2.2: Understanding Elasticsearch Data Structures

Before analyzing the document, let’s understand how Elasticsearch stores different types of data structures:

Single object:

1
2
3
4
5

"user": {
  "id": "U123",
  "email": "user@example.com",
  "country": "France"
}

A single object with properties. Accessible via dot notation: user.id, user.email, user.country, etc.

Array of simple values:

1

"tags": ["important", "verified", "priority"]

A list of simple values of the same type. Each value can be queried independently.

Array of objects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

"items": [
  {
    "id": "ITEM-001",
    "name": "Widget A",
    "type": "electronics",
    "price": 99.99
  },
  {
    "id": "ITEM-002",
    "name": "Gadget B",
    "type": "accessories",
    "price": 24.50
  }
]

A list of objects with properties.

Important: Arrays of objects (like products) have special behavior in Elasticsearch. We’ll explore this in detail in later workshops.