Course contentsShow
← Cloud and Infrastructure
Lesson 2011 of 2,98943. Observability - LoggingFree lesson

Log Indexing Fundamentals

Learn how logs are indexed for fast searching and the trade-offs between index size and query speed.

Log Indexing Fundamentals

What you'll learn: How logs are indexed to enable lightning-fast searches and the trade-offs you'll face between index size and query speed.


What Is Log Indexing?

When you aggregate millions of logs per day (remember Log Aggregation Architecture), searching through them linearly would be impossibly slow. Log indexing is the process of creating data structures that act like a book's index—allowing you to jump directly to relevant logs instead of reading every line.

Think of it like a library: without an index, you'd scan every book to find mentions of "Kubernetes errors." With an index, you instantly locate the shelf, book, and page.

How Indexing Works

After logs are parsed and enriched (from lesson 2010), indexing systems extract searchable terms—fields like level, timestamp, service_name, or keywords from the message. These terms are mapped to the log's storage location.

Common indexing strategies:

  • Full-text indexing: Every word becomes searchable (great for free-text searches, but expensive)
  • Field-based indexing: Only specific structured fields are indexed (faster, smaller, but limited scope)
  • Time-based partitioning: Logs split by time windows, dramatically speeding queries within a time range

The Trade-Offs

Larger indexes = faster queries, but:

  • More storage consumed (indexes can equal 50%+ of raw log size)
  • Higher memory requirements during searches
  • Slower log ingestion (indexing takes time)

Smaller/selective indexes = slower queries, but:

  • Lower storage costs
  • Faster writes and ingestion
  • Reduced operational overhead

Most teams balance these by indexing frequently-searched fields (like error, user_id, trace_id) while skipping verbose message bodies for cold storage.


Key Takeaway:
Log indexing transforms chaotic log streams into queryable data by creating lookup structures; you must balance index comprehensiveness against storage costs and ingestion speed to match your observability needs.