Course contentsShow
← System Design
Lesson 1165 of 1,91929. Observability: LoggingFree lesson

Log Retention Policies

Balance storage costs with compliance needs; tier logs by age, archiving old data to cheaper storage.

Log Retention Policies

What you'll learn: How to balance storage costs with compliance requirements by tiering logs based on age and archiving old data to cheaper storage.

Why Retention Policies Matter

As your system generates millions of logs daily, storing everything forever becomes prohibitively expensive and impractical. Yet you can't just delete logs arbitrarily—legal regulations, audit trails, and debugging needs may require specific retention periods. Log retention policies define how long to keep logs and where to store them as they age.

The Tiering Strategy

Think of logs like wine in a cellar: fresh logs (recent incidents) need to be immediately accessible on fast, expensive storage. Older logs are rarely accessed but must be kept for compliance—these can move to slower, cheaper "cold storage."

A typical tiered approach:

  • Hot tier (0-7 days): Fast SSD-backed search indexes (Elasticsearch, Splunk). Supports real-time queries and alerting.
  • Warm tier (8-90 days): Compressed storage with slower query performance. For occasional investigations.
  • Cold tier (91+ days): Object storage like S3 Glacier. Rarely queried, but preserved for audits or legal holds. Retrieval may take hours.
  • Deletion (after retention period): Permanently remove logs that exceed legal and business requirements.

Balancing Costs and Compliance

Define retention based on:

  • Compliance mandates: GDPR, HIPAA, or industry regulations may require 1–7 years.
  • Operational needs: How far back do you realistically investigate incidents? Often 30–90 days.
  • Storage budget: Cold storage costs ~1% of hot storage per GB.

Example: Keep application logs hot for 7 days, warm for 30 days, cold for 1 year (compliance), then delete. Security logs might stay cold for 7 years.

Key Takeaway: Log retention policies balance operational access speed, storage costs, and compliance requirements by tiering logs across hot, warm, and cold storage based on age and query patterns.