Course contentsShow
← Cloud and Infrastructure
Lesson 2020 of 2,98943. Observability - LoggingFree lesson

Elasticsearch Index Lifecycle Management

Automating log retention with ILM policies: hot, warm, cold, and delete phases.

Elasticsearch Index Lifecycle Management

What you'll learn: How to automate log retention using ILM policies that transition indices through hot, warm, cold, and delete phases.

Why ILM Matters

In centralized logging, old indices pile up quickly. Without automation, you'd manually delete stale data or watch storage costs skyrocket. Index Lifecycle Management (ILM) automates the entire lifecycle—from fresh, frequently-queried logs to eventual deletion—based on age or size triggers.

The Four Phases

Think of ILM like aging wine: logs start valuable and expensive, then move to cheaper storage as they age, and eventually get discarded.

Hot Phase

New logs land here. Indices are actively written to and frequently queried. They live on your fastest (and most expensive) storage—typically SSDs. You might rollover to a new index daily or when size hits 50GB.

Warm Phase

Logs that are days or weeks old. No longer written to, but still occasionally searched. ILM moves them to slower, cheaper storage and may reduce replica count to save space.

Cold Phase

Rarely queried logs (months old). Moved to the cheapest storage tier. Elasticsearch can use searchable snapshots here—data lives in object storage, loaded only when queried.

Delete Phase

After your retention period (e.g., 90 days), indices are automatically deleted. No manual cleanup scripts needed.

How It Works

You define an ILM policy in Elasticsearch with rules like "move to warm after 7 days" or "delete after 90 days." Then attach that policy to your index template. Elasticsearch monitors indices and executes transitions automatically in the background.

Key Takeaway: ILM policies automate log retention by transitioning indices through hot, warm, cold, and delete phases based on age or size, balancing query performance with storage costs without manual intervention.