In this lesson, you will learn how to balance observability costs and system performance by implementing strategic trace sampling patterns.
As you scale your microservices in Kubernetes, capturing 100% of your traces (full fidelity) quickly becomes prohibitively expensive in terms of network bandwidth, storage, and processing overhead. To solve this, we use sampling strategies to determine which traces to keep and which to discard.
Head-Based Sampling
The most common approach is head-based sampling, which makes a decision at the very start of a request. When an entry-point service (like an Ingress controller) receives a request, it randomly decides whether to trace that entire flow based on a fixed percentage.
- Pros: Extremely efficient; it requires zero communication between services since the "sampled" flag is passed along in the trace context.
- Cons: You might miss critical errors. If you sample at 1%, a rare but catastrophic 500-error might be discarded because it wasn't chosen by the head-based logic.
Tail-Based Sampling
To solve the shortcomings of head-based sampling, we use tail-based sampling. This strategy requires an OpenTelemetry Collector to buffer the spans of a trace until the entire request is complete. Only then does the collector look at the "tail" of the trace to decide whether to keep it.
This allows you to implement intelligent policies, such as:
- Always keep traces that contain an error.
- Always keep traces that exceed a specific latency threshold (e.g., > 500ms).
- Keep a small percentage (e.g., 0.1%) of successful, fast requests for baseline health.
You can configure this in your OpenTelemetry Collector config.yaml:
processors:
tail_sampling:
policies:
- name: errors-only
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-requests
type: latency
latency:
threshold_ms: 500
By buffering in the collector, you gain the "best of both worlds": you keep the signals that matter for debugging (the errors and slow spikes) while aggressively dropping the noise from healthy, high-traffic endpoints.
Key Takeaway: Use head-based sampling for high-traffic, stable services to save costs, and employ tail-based sampling in your collectors to ensure that critical performance anomalies and errors are never discarded.