Elasticsearch Index Lifecycle Management
What you'll learn: How to automate log retention using ILM policies that transition indices through hot, warm, cold, and delete phases.
Why ILM Matters
In centralized logging, old indices pile up quickly. Without automation, you'd manually delete stale data or watch storage costs skyrocket. Index Lifecycle Management (ILM) automates the entire lifecycle—from fresh, frequently-queried logs to eventual deletion—based on age or size triggers.
The Four Phases
Think of ILM like aging wine: logs start valuable and expensive, then move to cheaper storage as they age, and eventually get discarded.
Hot Phase
New logs land here. Indices are actively written to and frequently queried. They live on your fastest (and most expensive) storage—typically SSDs. You might rollover to a new index daily or when size hits 50GB.
Warm Phase
Logs that are days or weeks old. No longer written to, but still occasionally searched. ILM moves them to slower, cheaper storage and may reduce replica count to save space.
Cold Phase
Rarely queried logs (months old). Moved to the cheapest storage tier. Elasticsearch can use searchable snapshots here—data lives in object storage, loaded only when queried.
Delete Phase
After your retention period (e.g., 90 days), indices are automatically deleted. No manual cleanup scripts needed.
How It Works
You define an ILM policy in Elasticsearch with rules like "move to warm after 7 days" or "delete after 90 days." Then attach that policy to your index template. Elasticsearch monitors indices and executes transitions automatically in the background.
Key Takeaway: ILM policies automate log retention by transitioning indices through hot, warm, cold, and delete phases based on age or size, balancing query performance with storage costs without manual intervention.