Course contentsShow
← System Design
Lesson 1228 of 1,91931. Observability: TracingFree lesson

Trace Sampling Fundamentals

Why collecting every trace is prohibitively expensive and how sampling reduces overhead while retaining diagnostic value.

Trace Sampling Fundamentals

You'll learn why recording every single trace is unsustainable at scale and how sampling keeps tracing systems practical without losing critical diagnostic information.

The Cost Problem

Imagine your e-commerce platform handles 100,000 requests per second. If each trace generates just 10 spans with metadata, you're producing 1 million spans per second. Storing, transmitting, and analyzing every single one becomes prohibitively expensive:

  • Network overhead: Shipping spans to collectors saturates bandwidth
  • Storage costs: Petabytes of trace data accumulate quickly
  • Processing load: Analyzing every trace overwhelms backend systems
  • Application impact: Heavy instrumentation slows down services

At high traffic volumes, the observability system itself becomes a bottleneck.

How Sampling Solves This

Trace sampling means intentionally recording only a subset of traces—say 1% or 0.1%—rather than capturing everything. Think of it like polling: you don't need to survey every citizen to understand a population's opinion.

The key insight: patterns repeat. If 99% of your checkout flows succeed identically, recording all of them provides minimal additional value. But that 1% with errors? Those you need.

Sampling dramatically reduces:

  • Data volume (and storage costs)
  • Network bandwidth usage
  • Backend processing requirements
  • Impact on application performance

While retaining enough traces to:

  • Identify performance bottlenecks
  • Debug errors and anomalies
  • Understand system behavior
  • Calculate accurate aggregate metrics (like p99 latency)

Smart sampling strategies can even bias toward interesting traces—like those with errors or unusually high latencies—ensuring you don't lose the diagnostic value you care about most.

Key Takeaway: Trace sampling trades complete trace coverage for sustainable observability at scale, capturing enough representative data to diagnose issues without overwhelming your infrastructure or budget.