Log Retention and Storage
You'll learn how to organize logs into storage tiers based on how often you access them and how long you must keep them.
The Core Idea
Not all logs are created equal. Recent logs from the last few hours need lightning-fast access for troubleshooting active incidents. Logs from last month might be queried occasionally for trend analysis. Logs from two years ago? They're probably just sitting there for compliance audits—rarely touched but legally required.
Storage tiers let you match log age and access patterns to the right storage technology and cost:
Hot Storage
High-performance, expensive storage (SSD-backed databases, in-memory caches) for recent logs—typically the last 1-7 days. You query these constantly during debugging and real-time monitoring. Think of this as your "active workspace."
Warm Storage
Mid-tier storage (spinning disks, slower databases) for moderately old logs—often 7-90 days. You access these occasionally for historical analysis, performance reviews, or investigating patterns. This is your "filing cabinet."
Cold Storage
Cheap, archival storage (object storage like S3, Azure Blob, or tape) for ancient logs—months to years old. These exist mainly for compliance (GDPR, HIPAA, SOX) and are rarely queried. This is your "warehouse in another city."
Balancing Cost and Access
A common strategy: retain 7 days in hot storage for fast queries, move to warm for 30 days, then archive to cold for 1-7 years depending on regulations. This dramatically reduces costs—cold storage can be 50-100× cheaper than hot.
Key Takeaway: Design log retention policies around access frequency and compliance needs, using hot/warm/cold tiers to balance query performance against storage costs.