Course contentsShow
← System Design
Lesson 1254 of 1,91931. Observability: TracingFree lesson

Tail-Based Sampling

Sampling after trace completion based on characteristics like errors or latency; requires buffering spans.

Tail-Based Sampling

You'll learn how to intelligently sample traces after they complete, keeping only the most important ones based on actual behavior.


What Is Tail-Based Sampling?

Tail-based sampling is a tracing strategy where you delay the sampling decision until after a trace completes. Unlike head-based sampling (which decides at the start), tail-based sampling lets you examine the entire trace—its duration, error status, span attributes—before deciding whether to keep or discard it.

Think of it like a movie critic who watches the entire film before deciding whether to review it, rather than judging from just the opening scene.


Why It Matters

With head-based sampling, you might discard a trace that later encounters a critical error—exactly the trace you'd want to investigate. Tail-based sampling solves this by:

  • Keeping all error traces regardless of sampling rate
  • Capturing slow requests that exceed latency thresholds
  • Retaining traces with specific attributes (e.g., premium user IDs, critical transactions)

This ensures your most valuable debugging data survives sampling.


The Tradeoff: Buffering

The catch? You must buffer all spans in memory until the trace completes. Each span arriving from distributed services sits in a collector's buffer, waiting for siblings to arrive. Once the root span finishes, the collector evaluates the complete trace and decides: keep or drop?

This requires significant memory and adds complexity—you need timeout logic for incomplete traces and mechanisms to handle buffer pressure.


When to Use It

Tail-based sampling shines when:

  • Error rates are low but critical to capture
  • You need adaptive sampling based on trace characteristics
  • Storage costs are high, but you can't miss important traces

Key Takeaway: Tail-based sampling makes smarter retention decisions by evaluating complete traces, but requires buffering spans and careful resource management—use it when trace quality matters more than simplicity.