Course contentsShow
← System Design
Lesson 1255 of 1,91931. Observability: TracingFree lesson

Adaptive Sampling

Dynamically adjusting sample rates based on traffic volume, error rates, or other system signals.

Adaptive Sampling

You'll learn how to automatically adjust trace sampling rates in real-time based on changing system conditions.

The Core Idea

Adaptive sampling is a dynamic approach that continuously tunes sampling rates based on live system signals like traffic volume, error rates, or latency spikes. Instead of using a fixed sampling percentage (e.g., always sample 1% of traces), the system observes current conditions and increases sampling when problems arise or decreases it during high-volume, steady-state operation.

Think of it like a camera with auto-focus: it adjusts to capture what's important in the moment, rather than using a fixed lens setting regardless of the scene.

How It Works

The tracing system monitors key signals:

  • Traffic volume: Sample more aggressively during low traffic (you can afford it), less during peaks
  • Error rates: Increase sampling when errors spike to capture more failure traces
  • Latency patterns: Sample more slow requests to investigate performance degradation
  • Resource constraints: Reduce sampling if storage or processing capacity is stressed

The system periodically recalculates sampling rates—perhaps every few seconds or minutes—and adjusts the probability that new traces get sampled.

Why It Matters

Adaptive sampling balances two competing goals:

  1. Cost efficiency: Don't waste resources storing traces during normal operation
  2. Incident visibility: Capture more data exactly when you need it most

This is especially valuable when you combine it with tail-based sampling (already learned), where the adaptive decision can influence which completed traces to keep based on their characteristics.

Key Takeaway:
Adaptive sampling dynamically adjusts trace collection rates based on real-time system signals like errors and traffic, ensuring you capture critical data during incidents while controlling costs during normal operation.