Course contentsShow
← System Design
Lesson 1135 of 1,91929. Observability: LoggingFree lesson

Log Retention and Volume Management

Planning for log storage costs: determining retention policies based on compliance, debugging needs, and storage capacity.

Log Retention and Volume Management

What you'll learn: How to balance log storage costs with compliance, debugging needs, and infrastructure capacity through strategic retention policies.

The Storage Challenge

Logs accumulate fast. A moderately busy service generating 100 logs per second creates over 8.6 million logs daily. Without a retention strategy, storage costs spiral while finding relevant logs becomes impossible—like searching for a needle in an ever-growing haystack.

Planning Retention Policies

Compliance requirements often dictate minimum retention. Financial systems might need 7 years of audit logs, healthcare systems 6+ years under regulations like HIPAA. These aren't negotiable.

Debugging needs vary by log type:

  • Error logs: 30-90 days typically suffice for investigating recent incidents
  • Access logs: 7-30 days for security investigations
  • Debug logs: 3-7 days (expensive, high volume, rarely needed after immediate troubleshooting)

Storage tiering reduces costs:

  • Hot storage (0-7 days): Fast SSDs for active debugging
  • Warm storage (8-90 days): Slower, cheaper disks for occasional queries
  • Cold storage (90+ days): Archive systems like S3 Glacier for compliance

Volume Management Strategies

Sampling reduces volume without losing visibility—log 100% of errors but only 1% of successful requests. You see patterns without drowning in noise.

Aggregation replaces repetitive logs with summaries. Instead of logging every cache hit, emit metrics: "1000 cache hits in the last minute."

Compression typically achieves 5-10x space savings on text logs. Enable it immediately after log rotation.

The Cost Calculation

If you generate 10 TB of logs monthly at 0.023/GBforstorage,that′s0.023/GB for storage, that's 230/month—or $2,760 annually. Multiply across services and environments, and log storage becomes a significant budget line requiring active management.

Key Takeaway: Effective log retention balances compliance mandates, debugging windows, and storage costs through tiered retention policies, sampling strategies, and proactive volume management.