Course contentsShow
← System Design
Lesson 1253 of 1,91931. Observability: TracingFree lesson

Head-Based Sampling

Making sampling decisions at trace start; simple but can miss rare errors or slow requests that appear later.

Head-Based Sampling

What you'll learn: How sampling decisions made at the very start of a trace can simplify tracing systems but potentially miss important edge cases.

The Core Idea

Head-based sampling means making the decision to keep or discard a trace at the moment it begins—at the "head" or root span of the request. Once you decide to sample a trace (say, keep 10% of all requests), every span in that trace gets recorded. If you decide not to sample it, the entire trace is discarded.

Think of it like a bouncer at a club entrance: they decide who gets in at the door based on a simple rule (every 10th person), not by observing how interesting each person turns out to be once inside.

Why It's Simple

The beauty of head-based sampling is its simplicity:

  • The decision happens once, at trace start
  • No coordination needed between services mid-flight
  • Predictable overhead and storage costs
  • Easy to implement with a random number generator

When your root span begins, you generate a number and if it's within your sampling threshold, you propagate "yes, sample this" in the trace context to all downstream services.

The Critical Limitation

Here's the problem: you're making a blind decision before knowing what will happen. A trace that looks ordinary at the start might:

  • Encounter a rare database deadlock 5 services deep
  • Experience a slow third-party API call
  • Hit an edge-case error in a downstream system

Since you already decided "no" at the head, you never recorded any of it. You've missed the exact traces you most wanted to see—the slow ones and the broken ones.

Key Takeaway: Head-based sampling is straightforward and performant but can't adapt to what actually happens during request execution, making it prone to missing rare but critical traces.