1. Setup & Prerequisites

Duration30min AI Banned

Introduction

Before diving into Elasticsearch queries and data analysis, we need to ensure your development environment is properly configured. This section will guide you through deploying the ELK stack, verifying the infrastructure, and testing the data generator.

Think of this as preparing your laboratory before the experiment—we’ll make sure all the equipment is working before we start the real work.

Prerequisites

Before starting, ensure you have the following installed on your machine:

• ✅ Docker Desktop (includes Docker Engine + Docker Compose)
• ✅ uv package manager (manages Python automatically)
• ✅ curl (for testing HTTP endpoints - usually pre-installed)
• ✅ A web browser (Chrome, Firefox, or Safari)

Install uv Package Manager

Install uv (it will automatically manage Python for you):

1

curl -LsSf https://astral.sh/uv/install.sh | sh

After installation, restart your terminal.

Verify Installation

Check that the tools are properly installed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Check Docker Desktop
docker --version
# Expected: Docker version 20.10.0 or higher

docker-compose --version
# Expected: Docker Compose version 2.0.0 or higher

# Check uv
uv --version
# Expected: uv 0.x.x or higher

Step 1: Download the Workshop Environment

1.1 Download and Extract the Archive

Download the workshop environment (ZIP archive): elasticsearch-workshop.zip

1.2 Understanding the Workshop Structure

The extracted archive contains the following structure:

elk_workshop/
├── docker-compose.yml       # ELK stack orchestration
├── .env                     # Environment variables (ports, versions, memory)
├── logstash/
│   └── pipeline/
│       ├── 01-transactions.conf  # Transaction data pipeline
│       └── 02-applogs.conf       # Application logs pipeline
└── generators/
    ├── generator.py         # Main data generator
    ├── config.yaml          # Generator configuration
    ├── pyproject.toml       # Python dependencies
    ├── data_models.py       # Data structures
    └── lib/
        ├── ecommerce_generator.py
        └── scenario_manager.py

Key Components:

• docker-compose.yml: Defines the Elasticsearch, Logstash, and Kibana services
• .env: Configuration file for ports, ELK version, and memory settings
• logstash/pipeline/: Logstash configuration for ingesting transaction and log data
• generators/: Python-based data generator that simulates e-commerce transactions

Step 2: Deploy the ELK Stack

2.1 Start the Docker Services

The workshop includes a docker-compose.yml file that defines the entire ELK stack (Elasticsearch, Logstash, and Kibana).

1

docker-compose up -d

Explanation:

• up: Start the services
• -d: Run in detached mode (background)

2.2 Verify Services Are Running

Check that all three services are up and healthy:

1

docker-compose ps

Expected Output:

NAME                    STATUS          PORTS
elk_workshop-elasticsearch-1   Up          0.0.0.0:9200->9200/tcp
elk_workshop-logstash-1        Up          0.0.0.0:5000-5001->5000-5001/tcp
elk_workshop-kibana-1          Up          0.0.0.0:5601->5601/tcp

All services should show STATUS: Up.

1
2
3

docker-compose logs elasticsearch
docker-compose logs logstash
docker-compose logs kibana

2.3 Wait for Services to be Ready

Even when services show “Up”, they may still be initializing. Wait for them to be fully ready:

Check Elasticsearch:

1

curl http://localhost:9200/_cluster/health

Expected Output (after 30-60 seconds):

1
2
3
4
5
6

{
  "cluster_name": "docker-cluster",
  "status": "yellow",
  "number_of_nodes": 1,
  "number_of_data_nodes": 1
}

Status meaning:

• green: All primary and replica shards allocated (ideal)
• yellow: All primary shards allocated, but some replicas missing (normal for single-node cluster)
• red: Some primary shards not allocated (problem!)

Check Kibana:

1

curl http://localhost:5601/api/status

Wait until you get a 200 OK response (may take 1-2 minutes).

Access Kibana UI:

Open your browser and navigate to: http://localhost:5601

You should see the Kibana welcome screen.

Step 3: Test the Data Generator

The workshop includes a Python-based e-commerce transaction generator. Let’s verify it works correctly before the exercises.

3.1 Navigate to the Generator Directory

1

cd generators

3.2 Install Dependencies

Use uv to install the required Python packages:

1

uv sync

What this does:

• Creates a virtual environment (.venv)
• Installs dependencies from pyproject.toml
• Locks versions in uv.lock

3.3 Run a 15-Second Test

Start the generator for a short test run:

1
2
3
4

uv run python generator.py &
GENERATOR_PID=$!
sleep 15
kill $GENERATOR_PID

Explanation:

• &: Run generator in background
• GENERATOR_PID=$!: Store the process ID
• sleep 15: Wait 15 seconds
• kill $GENERATOR_PID: Stop the generator

Expected Output:

╔════════════════════════════════════════════╗
║   ELK Workshop - Data Generator v2.0       ║
╚════════════════════════════════════════════╝

Configuration:
  Rate: 10 events/second
  Logstash: localhost:5000, 5001

Running in normal mode (no scenarios)

Press CTRL-C to stop

==================================================

✓ Connected to Logstash

Generated: 126 events | Rate: 12.5/sec | Elapsed: 10s

If you see “✓ Connected to Logstash” and events being generated, the generator is working correctly!

3.4 Verify Data Arrived in Elasticsearch

First, list all indices to see what was created:

1

curl http://localhost:9200/_cat/indices?v

Expected Output:

health status index                   uuid   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   transactions-2025.11.12 abc123  1   1        150            0     45.2kb         45.2kb
yellow open   logs-2025.11.12         def456  1   1         78            0     32.1kb         32.1kb

You should see indices with names like:

• transactions-YYYY.MM.DD - Transaction data
• logs-YYYY.MM.DD - Application logs

Count the transactions:

Now that you know the index name, count the documents:

1

curl http://localhost:9200/transactions-*/_count

Expected Output:

1
2
3
4
5
6
7
8

{
  "count": 150,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  }
}

Step 4: Explore Kibana

Now that data is flowing, let’s verify you can query it through Kibana.

4.1 Open Kibana Dev Tools

1. Open Kibana: http://localhost:5601
2. Click the ☰ menu (top left)
3. Navigate to Management → Dev Tools

Alternatively, go directly to: http://localhost:5601/app/dev_tools#/console

4.2 Run Your First Query

In the Dev Tools console, you’ll see a text editor. This is where you can send queries directly to Elasticsearch.

Try this query:

1
2
3
4

GET /transactions-*/_search
{
  "size": 1
}

To execute:

• Click the ▶️ Play button (green triangle), or
• Press Ctrl+Enter (Cmd+Enter on Mac)

Expected Response:

You should see a JSON response with:

• "took": Query execution time in milliseconds
• "hits.total.value": Total number of documents (around 150)
• "hits.hits": Array with one transaction document

4.3 Create a Data View and Explore with Discover

Now that you’ve learned to query data programmatically with Dev Tools, let’s explore Kibana’s visual interface. To access Analytics → Discover, you first need to create a data view.

What is a Data View?

A data view tells Kibana which indices to visualize and which field contains the timestamp for time-based filtering. Think of it as a “window” into your Elasticsearch indices.

Steps to Create a Data View:

1. In Kibana, click the ☰ menu (top left)
2. Navigate to Management → Stack Management
3. Under the “Kibana” section, click Data Views
4. Click the Create data view button
5. Fill in the form:
   • Name: Transactions
   • Index pattern: transactions-*
   • Timestamp field: Select @timestamp from the dropdown
6. Click Save data view to Kibana

Access Discover:

Once your data view is created:

1. Navigate to ☰ Menu → Analytics → Discover
2. You should now see your transaction data displayed visually with:
   • A timeline histogram showing event distribution over time
   • A field list on the left for filtering and analysis
   • Sample documents in the main panel

Discover vs Dev Tools:

• Discover: Great for visual exploration, spotting patterns, and building dashboards
• Dev Tools: Perfect for precise queries, aggregations, and programmatic control

Both tools complement each other—use Discover to explore, and Dev Tools to refine your queries!

Step 5: Clean Up Test Data (Optional)

If you want to start the workshop with a clean slate, delete the test data and replace YYYY-MM-DD with proper value.

1

curl -X DELETE http://localhost:9200/transactions-YYYY-MM-DD

Or in Kibana Dev Tools:

1

DELETE /transactions-YYYY-MM-DD

Response:

1
2
3

{
  "acknowledged": true
}

If during the workshop you want to delete all the data associated with a docker container, remember you need to delete the volumes : docker-compose down -v.

Summary

At this point, you should have:

• ✅ ELK stack running (Elasticsearch, Logstash, Kibana)
• ✅ Verified Elasticsearch cluster is healthy (yellow status)
• ✅ Successfully run the data generator for 15 seconds
• ✅ Confirmed transactions indexed in Elasticsearch
• ✅ Executed your first query in Kibana Dev Tools
• ✅ Created a data view and explored data with Discover