Log Retention and Volume Management
What you'll learn: How to balance log storage costs with compliance, debugging needs, and infrastructure capacity through strategic retention policies.
The Storage Challenge
Logs accumulate fast. A moderately busy service generating 100 logs per second creates over 8.6 million logs daily. Without a retention strategy, storage costs spiral while finding relevant logs becomes impossible—like searching for a needle in an ever-growing haystack.
Planning Retention Policies
Compliance requirements often dictate minimum retention. Financial systems might need 7 years of audit logs, healthcare systems 6+ years under regulations like HIPAA. These aren't negotiable.
Debugging needs vary by log type:
- Error logs: 30-90 days typically suffice for investigating recent incidents
- Access logs: 7-30 days for security investigations
- Debug logs: 3-7 days (expensive, high volume, rarely needed after immediate troubleshooting)
Storage tiering reduces costs:
- Hot storage (0-7 days): Fast SSDs for active debugging
- Warm storage (8-90 days): Slower, cheaper disks for occasional queries
- Cold storage (90+ days): Archive systems like S3 Glacier for compliance
Volume Management Strategies
Sampling reduces volume without losing visibility—log 100% of errors but only 1% of successful requests. You see patterns without drowning in noise.
Aggregation replaces repetitive logs with summaries. Instead of logging every cache hit, emit metrics: "1000 cache hits in the last minute."
Compression typically achieves 5-10x space savings on text logs. Enable it immediately after log rotation.
The Cost Calculation
If you generate 10 TB of logs monthly at 230/month—or $2,760 annually. Multiply across services and environments, and log storage becomes a significant budget line requiring active management.
Key Takeaway: Effective log retention balances compliance mandates, debugging windows, and storage costs through tiered retention policies, sampling strategies, and proactive volume management.