Course contentsShow
← System Design
Lesson 1150 of 1,91929. Observability: LoggingFree lesson

The ELK Stack: Elasticsearch

How Elasticsearch provides distributed full-text search and indexing for log data at scale.

The ELK Stack: Elasticsearch

What You'll Learn

You'll understand how Elasticsearch acts as the powerhouse search and indexing engine that makes massive volumes of log data queryable in near real-time.

The Challenge of Searching Billions of Logs

Once you've shipped logs centrally (via agents or direct shipping), you face a problem: how do you search through millions or billions of log entries efficiently? Reading files sequentially is impossibly slow, and traditional databases aren't optimized for full-text search across unstructured data.

Elasticsearch: A Distributed Search Engine

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It's designed specifically for indexing and searching massive volumes of data—perfect for logs.

Think of it like Google for your logs: instead of manually scanning files, Elasticsearch pre-indexes every field in your structured logs (like timestamp, service_name, error_code) so you can instantly query patterns like "show me all ERROR logs from service X in the last hour."

How It Works at Scale

Elasticsearch achieves scale through:

  • Inverted indexes: It creates lookup tables mapping terms to document locations (like a book's index)
  • Sharding: Data splits across multiple nodes horizontally; each shard is an independent Lucene index
  • Replication: Each shard has replicas for fault tolerance and read scalability
  • RESTful API: You query using simple JSON over HTTP

When you search level:ERROR AND service:checkout, Elasticsearch distributes the query across all shards in parallel, aggregates results, and returns matches in milliseconds—even across terabytes of logs.

The "E" in ELK

Elasticsearch is the storage and retrieval layer. Logstash (the "L") processes and ships logs into it, while Kibana (the "K") provides visualization on top—but Elasticsearch is where the magic of fast, scalable search happens.

Key Takeaway: Elasticsearch provides distributed full-text search by indexing log data across shards, enabling sub-second queries over massive datasets that would be impossible to search linearly.