Course contentsShow
← Cloud and Infrastructure
Lesson 2018 of 2,98943. Observability - LoggingFree lesson

Elasticsearch Fundamentals for Logging

Learn Elasticsearch core concepts: indices, documents, shards, and replicas for log storage.

Elasticsearch Fundamentals for Logging

You'll learn the core building blocks of Elasticsearch that enable fast, scalable log storage and retrieval.

What Is Elasticsearch?

Elasticsearch is a distributed search and analytics engine that stores logs as documents organized into indices. Think of it like a library: indices are sections (fiction, history), documents are individual books, and Elasticsearch provides a lightning-fast card catalog.

Core Concepts

Indices

An index is a logical collection of similar documents—for application logs, you might have one index per day like logs-2024-01-15. Indices let you organize and query related logs together efficiently.

Documents

A document is a single JSON object representing one log entry. It contains all the fields from your log event: timestamp, severity, message, hostname, etc. Documents are the fundamental unit Elasticsearch stores and searches.

Shards

Each index is split into shards—smaller pieces distributed across your cluster. Think of shards like splitting a massive file cabinet across multiple rooms. This enables:

  • Parallel processing (multiple nodes search simultaneously)
  • Horizontal scaling (add more nodes, distribute more shards)

A typical setup might split a daily log index into 3-5 primary shards.

Replicas

Replicas are copies of your primary shards stored on different nodes. They provide:

  • High availability (if a node fails, replicas serve requests)
  • Read scalability (queries can hit replicas, not just primaries)

With 1 replica, each primary shard has one backup copy on another node.

How They Work Together

When you send logs to Elasticsearch, it routes each document to the correct index, assigns it to a specific primary shard (based on document ID hash), and replicates it. Queries search across all shards in parallel, merging results almost instantly.

Key Takeaway: Elasticsearch stores logs as JSON documents in indices, splits indices into shards for distribution, and replicates shards for resilience—creating a fast, scalable logging backend.