3. Queries & Filtering

Duration45min AI Banned

Introduction

Now that you understand the Elastic Stack architecture and document structure, it’s time to master querying and filtering data. In this workshop, you’ll learn how to precisely search for documents, filter results, and understand the critical difference between text and keyword fields.

By the end of this session, you’ll be able to write production-ready queries that efficiently retrieve exactly the data you need.

Learning Objectives

By completing this workshop, you will be able to:

• ✅ Query documents using Elasticsearch Query DSL (term, range, bool queries)
• ✅ Filter and sort results using source filtering, sorting, and pagination
• ✅ Understand the difference between text and keyword field types
• ✅ Apply the correct query type for different use cases

Understanding Elasticsearch Queries

Before diving into exercises, let’s understand how Elasticsearch queries work. An Elasticsearch query is composed of several optional sections that control which data is returned and how it’s presented.

Query Anatomy

Here’s the general structure with all parameters used in this workshop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

GET /index-pattern/_search
{
  "_source": ["field1", "field2"],        // Source filtering: which fields to return

  "query": {                              // Query context: filter documents
    "bool": {                             // Boolean query: combine multiple conditions
      "must": [       // Conditions that MUST match (AND logic, affects score)
        { "term": { "status": "confirmed" } },  // Term query: exact match on keyword fields
        { "match": { "name": "laptop" } } // Match query: full-text search on text fields
      ],
      "should": [    // Conditions where at least one SHOULD match (OR logic, affects score)
        { "term": { "category": "electronics" } },
        { "term": { "category": "accessories" } }
      ],
      "filter": [                         // Conditions that MUST match (no scoring, faster)
        { "range": { "amount": { "gte": 100, "lte": 500 } } }  // Range query: numeric/date ranges
      ],
      "must_not": [                       // Conditions that MUST NOT match (exclusion, no scoring)
        { "term": { "status": "cancelled" } }
      ]
    }
  },

  "sort": [                              // Sort results
    { "timestamp": "desc" },             // By timestamp descending
    { "amount": "asc" }                  // Then by amount ascending
  ],

  "size": 10,                           // Number of documents to return (default: 10)
  "from": 0                             // Starting position (0 = first result, 10 = skip first 10)
}

Understanding Boolean Query Clauses:

The bool query has four types of clauses, each with different behavior:

• must: Documents MUST match these conditions (AND logic). Contributes to relevance score. Use for conditions that are required AND where scoring matters.

• should: At least one condition SHOULD match (OR logic). Contributes to relevance score. Documents matching more should clauses rank higher. If no must or filter exists, at least one should must match. If must or filter exists, should clauses are optional but boost score.

• filter: Documents MUST match these conditions (AND logic). Does NOT contribute to score. Faster than must because results are cached. Use for exact filters like ranges, statuses, or IDs where scoring doesn’t matter.

• must_not: Documents MUST NOT match these conditions (exclusion). Does NOT contribute to score. Use to exclude unwanted documents.

Other Key Concepts:

• _source filtering: Controls which fields are returned in results. Use ["field1", "field2"] to return specific fields, or false to exclude all source data.

• from and size (pagination): from specifies how many results to skip from the beginning of the result set (sorted by relevance or your sort clause). size specifies how many results to return. Example: from: 20, size: 10 returns results 21-30.

• term vs match: term queries perform exact matching (use for IDs, statuses, exact values). match queries perform full-text search with analysis (use for searching text content).

Response Anatomy

Every Elasticsearch search response has this structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

{
  "took": 4,                           // Query execution time in milliseconds
  "timed_out": false,                  // Whether the query timed out
  "_shards": {                         // Shard information
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {                            // Results container
    "total": {
      "value": 5832,                   // Total matching documents
      "relation": "eq"                 // Exact count or lower bound
    },
    "max_score": null,                 // Highest relevance score (null when sorting)
    "hits": [                          // Array of returned documents
      {
        "_index": "transactions-2025.12.01",  // Source index
        "_id": "Wn1a2poBLywqLnmDoP7h",        // Document ID
        "_score": null,                       // Relevance score (null when sorting)
        "_source": {                          // Actual document content
          // Your document fields here
        },
        "sort": [1764599963770]              // Sort values (when using sort)
      }
    ]
  }
}

Important fields:

• took: Query execution time - useful for performance monitoring
• hits.total.value: Total matching documents - may be limited to 10,000 by default
• hits.hits[]: Actual documents returned (limited by size parameter)
• _score: Relevance score - null when using sort or filter context

For detailed documentation on all query types, search modifiers, aggregations, and query contexts, refer to the Elasticsearch Documentation page.

Section 1: Basic Search Queries

Now let’s practice filtering transactions based on specific criteria. We’ll learn each query type and immediately apply it.

Exercise 1.1: Term Query - Exact Matching

Use term queries for exact matching on keyword fields (statuses, IDs, categories):

1
2
3
4
5
6
7

{
  "query": {
    "term": {
      "status.keyword": "confirmed"
    }
  }
}

Important: Use the .keyword subfield for exact matching on text fields. The text field without .keyword is analyzed (tokenized, lowercased), while .keyword preserves the exact value.

Exercise 1.2: Range Query - Numeric and Date Ranges

Use range queries for numeric values or date ranges:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "query": {
    "range": {
      "total_amount": {
        "gte": 100,    // greater than or equal
        "lte": 500     // less than or equal
      }
    }
  }
}

Available operators: gte (≥), lte (≤), gt (>), lt (<)

Exercise 1.3: Date Math - Time-Based Queries

For time-based queries, use date math expressions with range queries:

1
2
3
4
5
6
7
8
9

{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1h"    // last hour
      }
    }
  }
}

Common expressions:

• "now-1h" - last hour
• "now-1d" - last day
• "now-7d" - last 7 days
• "now-7d/d" - last 7 days rounded to day

Exercise 1.4: Boolean Query - Combining Conditions

Use bool queries to combine multiple conditions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "query": {
    "bool": {
      "must": [
        { "term": { "status.keyword": "confirmed" } },
        { "range": { "total_amount": { "gt": 200 } } }
      ]
    }
  }
}

The must clause ensures ALL conditions match (AND logic). Refer to the Query Anatomy section for other clause types (should, filter, must_not).

Section 2: Understanding Text vs Keyword

Now that you’ve written several queries using .keyword, let’s understand WHY this is necessary. This is one of the most important concepts in Elasticsearch.

Concepts: Text vs Keyword Field Types

Elasticsearch provides two main string field types with fundamentally different behaviors:

Text Fields (text type):

• Purpose: Full-text search (finding documents that contain certain words or phrases)
• Analysis process: Text is tokenized, lowercased, and stemmed before indexing
  • Example: “Wireless Bluetooth Headphones” → tokens: ["wireless", "bluetooth", "headphones"]
• Query behavior: Search terms are also analyzed, so “bluetooth” matches “Bluetooth”
• Use cases: Article content, product descriptions, user comments, any searchable text
• Limitations: Cannot be used for exact matching, sorting, or aggregations

Keyword Fields (keyword type):

• Purpose: Exact matching, filtering, sorting, and aggregations
• Analysis process: No analysis—the entire string is stored as-is
  • Example: “Wireless Bluetooth Headphones” → stored exactly as "Wireless Bluetooth Headphones"
• Query behavior: Must match exactly (case-sensitive)
• Use cases: IDs, email addresses, statuses, categories, tags, country codes
• Limitations: Not suitable for full-text search of long text

The Multi-Field Pattern (.keyword subfield):

Dynamic mapping often creates both types for string fields:

1
2
3
4
5
6
7
8
9

"product_name": {
  "type": "text",              // For full-text search
  "fields": {
    "keyword": {                // For exact matching/aggregations
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

This allows flexibility but consumes more storage (data is indexed twice).

Exercise 2.1: Testing Text vs Keyword Behavior

Test the difference between text and keyword behavior by running these three queries on the transactions-* index:

Query A: Using match on text field

1
2
3
4
5
6
7
8

GET /transactions-*/_search
{
  "query": {
    "match": {
      "products.name": "Bluetooth"
    }
  }
}

Query B: Using term on text field

1
2
3
4
5
6
7
8

GET /transactions-*/_search
{
  "query": {
    "term": {
      "products.name": "Bluetooth"
    }
  }
}

Query C: Using term on keyword subfield

1
2
3
4
5
6
7
8

GET /transactions-*/_search
{
  "query": {
    "term": {
      "products.name.keyword": "Wireless Bluetooth Headphones"
    }
  }
}