4. Mapping & Field Types

Duration45min AI Banned

Introduction

In the previous workshops, you’ve explored querying and filtering data in Elasticsearch. But have you wondered how Elasticsearch knows which fields are text, which are numbers, and how it stores arrays of objects?

In this workshop, you’ll dive into mapping—Elasticsearch’s schema definition. You’ll discover how Elasticsearch automatically mapped your transaction data and understand a critical limitation when querying arrays of objects.

By the end of this session, you’ll understand:

• How Elasticsearch stores different data types
• What dynamic mapping creates automatically
• The difference between object and nested types
• Why some queries on arrays return unexpected results

Learning Objectives

By completing this workshop, you will be able to:

• ✅ Examine automatically generated mappings using the Get Mapping API
• ✅ Understand how Elasticsearch detects and maps different field types
• ✅ Identify the difference between object and nested types for arrays
• ✅ Recognize when array queries return false positives

Section 1: Understanding Field Types & Mapping

Before diving into data analysis, you need to understand how Elasticsearch automatically mapped your transaction data. When the first document was indexed, Elasticsearch analyzed its structure and created a mapping based on the field types it detected.

Concepts: Dynamic Mapping

When you index a document without specifying a mapping, Elasticsearch uses dynamic mapping to automatically detect field types:

• String detection: Maps strings as both text (for full-text search) and keyword (for exact matching)
• Numeric detection: Maps numbers as appropriate numeric types (long, double, etc.)
• Date detection: Attempts to detect and parse date strings into date types
• Object detection: Nested JSON objects become object type by default

While convenient for getting started quickly, dynamic mapping often creates suboptimal structures:

• Creates unnecessary indices for fields that don’t need full-text search (storing data twice)
• Uses default settings that may not fit your use case
• May incorrectly interpret field types (e.g., numeric IDs as numbers instead of keywords)
• Doesn’t preserve array element correlation (uses object instead of nested)

Understanding dynamic mapping helps you identify optimization opportunities and design better explicit mappings.

Documentation: Get Mapping API

Exercise 1.1: Examine the Dynamic Mapping

Analyze the output:

1. Retrieve the current index mapping for one of the transaction indices
2. Analyze the field types for string fields (order_id, status, product names)
3. Identify how numeric fields (price, total_amount) were mapped
4. Examine how date fields (timestamp) were detected
5. Look at object/nested fields (customer, products) structure

Section 2: Understanding Object & Nested Types

Your transaction documents contain arrays of products. Understanding how Elasticsearch stores and queries these arrays is critical for accurate data analysis.

Concepts - Object vs Nested Types

When documents contain arrays of objects, Elasticsearch offers two different storage approaches with fundamentally different query behaviors.

Consider a simplified example with product reviews:

1
2
3
4
5
6
7

{
  "product_id": "PROD-123",
  "reviews": [
    {"author": "Alice", "rating": 5},
    {"author": "Bob", "rating": 2}
  ]
}

Object Type (default):

Elasticsearch flattens the array, storing fields separately:

1
2
3
4

{
  "reviews.author": ["Alice", "Bob"],
  "reviews.rating": [5, 2]
}

The problem: The connection between “Alice” and “5” is lost! A query for “reviews by Alice with rating 2” would incorrectly match this document (Alice from first review, rating 2 from second review).

Nested Type:

Elasticsearch stores each array element as a separate hidden document, preserving the relationship between fields within each review:

• Hidden doc 1: author=“Alice”, rating=5
• Hidden doc 2: author=“Bob”, rating=2

The benefit: A nested query for “reviews by Alice with rating 2” correctly rejects this document.

Trade-off: Nested types use more storage and require special query syntax, but provide accurate results for correlated queries.

Documentation:

• Object field type
• Nested field type

Exercise 2.1: Object Type Limitation

Let’s experience this limitation firsthand with the products array in your transaction data.

Exercise 2.2: When Does This Matter?

Understanding when the object type limitation matters is crucial for data engineering decisions.

Section 3: The Solution - Nested Type

Now that you understand the problem, let’s briefly understand the solution.

Concepts: How Nested Type Solves the Problem

To solve the object array limitation, the products field needs to be mapped as nested type instead of the default object type.

How nested type works:

1. Each array element is stored as a separate hidden document
2. The relationship between fields within each element is preserved
3. Special nested query syntax is required to access nested documents
4. Elasticsearch internally joins parent document with nested documents during queries

Example nested mapping:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "mappings": {
    "properties": {
      "products": {
        "type": "nested",
        "properties": {
          "product_id": { "type": "keyword" },
          "name": { "type": "text" },
          "category": { "type": "keyword" },
          "base_price": { "type": "float" }
        }
      }
    }
  }
}

Nested query syntax:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "query": {
    "nested": {
      "path": "products",
      "query": {
        "bool": {
          "must": [
            { "term": { "products.category": "Electronics" } },
            { "range": { "products.base_price": { "lt": 100 } } }
          ]
        }
      }
    }
  }
}

The nested query wrapper ensures that both conditions are evaluated on the same nested document (same product), preventing false positives.

Exercise 3.1: Understanding the Trade-offs

Let’s think critically about when to use nested types.