Log Retention Policies
What you'll learn: How to balance storage costs with compliance requirements by tiering logs based on age and archiving old data to cheaper storage.
Why Retention Policies Matter
As your system generates millions of logs daily, storing everything forever becomes prohibitively expensive and impractical. Yet you can't just delete logs arbitrarily—legal regulations, audit trails, and debugging needs may require specific retention periods. Log retention policies define how long to keep logs and where to store them as they age.
The Tiering Strategy
Think of logs like wine in a cellar: fresh logs (recent incidents) need to be immediately accessible on fast, expensive storage. Older logs are rarely accessed but must be kept for compliance—these can move to slower, cheaper "cold storage."
A typical tiered approach:
- Hot tier (0-7 days): Fast SSD-backed search indexes (Elasticsearch, Splunk). Supports real-time queries and alerting.
- Warm tier (8-90 days): Compressed storage with slower query performance. For occasional investigations.
- Cold tier (91+ days): Object storage like S3 Glacier. Rarely queried, but preserved for audits or legal holds. Retrieval may take hours.
- Deletion (after retention period): Permanently remove logs that exceed legal and business requirements.
Balancing Costs and Compliance
Define retention based on:
- Compliance mandates: GDPR, HIPAA, or industry regulations may require 1–7 years.
- Operational needs: How far back do you realistically investigate incidents? Often 30–90 days.
- Storage budget: Cold storage costs ~1% of hot storage per GB.
Example: Keep application logs hot for 7 days, warm for 30 days, cold for 1 year (compliance), then delete. Security logs might stay cold for 7 years.
Key Takeaway: Log retention policies balance operational access speed, storage costs, and compliance requirements by tiering logs across hot, warm, and cold storage based on age and query patterns.