Sampling Strategies Overview
You'll learn why capturing every single trace is impractical and how sampling controls costs.
The Tracing Cost Problem
In a high-traffic distributed system, tracing every single request creates overwhelming overhead. Imagine a service handling 100,000 requests per second—if each trace generates 50 spans with metadata, you're producing 5 million spans every second. This translates to:
- Network bandwidth flooding your trace collectors
- Storage costs exploding as you retain billions of spans
- Query performance degrading as your trace backend struggles with massive datasets
- Agent overhead consuming precious CPU and memory on production hosts
Tracing everything becomes prohibitively expensive at scale.
What is Sampling?
Sampling is the practice of selectively recording only a subset of traces rather than all of them. Think of it like a political poll—you don't survey every voter to understand trends; a representative sample gives you the insights you need at a fraction of the cost.
With sampling, you might trace:
- 1% of normal traffic (capturing patterns and typical behavior)
- 100% of errors (preserving debugging data for failures)
- 100% of slow requests (identifying performance bottlenecks)
This drastically reduces data volume while maintaining observability into system behavior.
Why Sampling Matters
You've already learned about Span Sampling Decisions (lesson 1238)—the mechanism that determines whether to record a trace. Sampling strategies are the policies that guide those decisions: when to sample, how much to sample, and what criteria to use.
Without thoughtful sampling, you either blow your observability budget or lose critical debugging data.
Key Takeaway: Sampling is essential for controlling tracing costs at scale—it lets you balance observability value against resource consumption by selectively recording representative traces rather than capturing everything.