Elasticsearch Fundamentals for Logging
You'll learn the core building blocks of Elasticsearch that enable fast, scalable log storage and retrieval.
What Is Elasticsearch?
Elasticsearch is a distributed search and analytics engine that stores logs as documents organized into indices. Think of it like a library: indices are sections (fiction, history), documents are individual books, and Elasticsearch provides a lightning-fast card catalog.
Core Concepts
Indices
An index is a logical collection of similar documents—for application logs, you might have one index per day like logs-2024-01-15. Indices let you organize and query related logs together efficiently.
Documents
A document is a single JSON object representing one log entry. It contains all the fields from your log event: timestamp, severity, message, hostname, etc. Documents are the fundamental unit Elasticsearch stores and searches.
Shards
Each index is split into shards—smaller pieces distributed across your cluster. Think of shards like splitting a massive file cabinet across multiple rooms. This enables:
- Parallel processing (multiple nodes search simultaneously)
- Horizontal scaling (add more nodes, distribute more shards)
A typical setup might split a daily log index into 3-5 primary shards.
Replicas
Replicas are copies of your primary shards stored on different nodes. They provide:
- High availability (if a node fails, replicas serve requests)
- Read scalability (queries can hit replicas, not just primaries)
With 1 replica, each primary shard has one backup copy on another node.
How They Work Together
When you send logs to Elasticsearch, it routes each document to the correct index, assigns it to a specific primary shard (based on document ID hash), and replicates it. Queries search across all shards in parallel, merging results almost instantly.
Key Takeaway: Elasticsearch stores logs as JSON documents in indices, splits indices into shards for distribution, and replicates shards for resilience—creating a fast, scalable logging backend.