Course contentsShow
← System Design
Lesson 1156 of 1,91929. Observability: LoggingFree lesson

Indexing Strategies and Retention

Managing index size, hot-warm-cold storage tiers, and retention policies to control costs.

Indexing Strategies and Retention

What you'll learn: How to manage log storage costs by strategically indexing data and moving it through storage tiers based on age and access patterns.

The Storage Cost Problem

Once you're shipping logs to a centralized system like Elasticsearch or Splunk, you'll quickly discover that storage costs explode. Full-text indexing every field in every log for instant search is expensive. Keeping years of detailed logs online is even more expensive.

Hot-Warm-Cold Storage Tiers

Think of your closet: everyday clothes stay within arm's reach (hot), seasonal items go to a higher shelf (warm), and old memorabilia moves to the attic (cold).

Hot tier: Recent logs (hours to days old) stored on fast SSDs with full indexing. You query these constantly for debugging active incidents.

Warm tier: Older logs (days to weeks) moved to slower, cheaper storage. Still indexed and searchable, but queries take longer. Perfect for compliance reviews or investigating patterns.

Cold tier: Ancient logs (months to years) archived to object storage like S3. Minimal or no indexing. Expensive to search but cheap to store. You only touch these for audits or legal holds.

Selective Indexing

You don't need to index every field. Index timestamp, service_name, level, and correlation_id for fast filtering. Leave detailed stack traces and message bodies as stored-only fields that appear in results but aren't searchable.

Retention Policies

Define rules: "Keep debug logs for 7 days, info logs for 30 days, error logs for 90 days." Delete data automatically when its retention period expires. This prevents your storage from growing forever while keeping critical error history.

Key Takeaway: Control logging costs by tiering storage based on age and access patterns, selectively indexing high-value fields, and automatically deleting logs after their retention period expires.