Course contentsShow
← System Design
Lesson 1246 of 1,91931. Observability: TracingFree lesson

Trace Data Retention Policies

Design retention strategies balancing storage costs with the need to debug historical issues across different trace ages.

Trace Data Retention Policies

What you'll learn: How to design retention strategies that balance storage costs with the ability to investigate historical production issues.

The Challenge of Trace Storage

Distributed traces are expensive to store. A single user request might generate dozens of spans with rich metadata, tags, and timing information. At scale, this creates terabytes of data daily. Yet you need traces available when investigating incidents that might have started days or weeks ago.

Designing a Tiered Retention Strategy

Think of trace retention like organizing photos: you keep recent ones in easy reach, archive important events longer, and eventually delete routine snapshots.

Hot storage (0-7 days): Keep all sampled traces with full fidelity. This covers most debugging scenarios—production issues are usually caught within days. Store in fast, queryable backends like your Jaeger or Zipkin storage.

Warm storage (7-30 days): Retain only high-value traces: errors, slow requests (P99 latency), or specific user-flagged transactions. Compress data and potentially move to cheaper storage like S3. You can still query these, just slower.

Cold/archived (30-90 days): Keep only trace metadata and aggregated statistics for compliance or trend analysis. Full span details are deleted.

Deleted (90+ days): Unless regulatory requirements demand it, purge everything.

Cost vs Value Trade-offs

Storage isn't just disk space—it's indexing overhead, query performance, and backup costs. A common pattern: sample aggressively (1-5% of normal traffic), but always capture 100% of errors. This ensures you can debug failures without drowning in data from successful requests.

Configure retention based on trace characteristics using span attributes you learned earlier: retain all traces with error=true, or those exceeding your critical path analysis thresholds.

Key Takeaway: Effective trace retention uses tiered storage based on trace age and value, keeping recent and error traces accessible while aggressively pruning routine successful requests to control costs.