Sampling for High-Volume Logs
You'll learn how to drastically cut logging costs and storage by selectively recording only a fraction of high-volume debug logs while keeping all errors visible.
The Core Idea
When a service handles millions of requests per second, logging every single request at DEBUG or INFO level becomes prohibitively expensive—both in storage costs and ingestion overhead. Sampling means intentionally recording only a subset of these verbose logs (say, 1% or 0.1%), while still logging all errors and warnings at full rate.
Think of it like quality control in manufacturing: you don't inspect every single widget coming off the assembly line—you sample periodically to monitor normal operation. But if a widget breaks, you always log that failure.
Why Sample?
- Cost reduction: Storing 1% of debug logs instead of 100% can reduce your logging bill by 99%
- Performance: Less I/O and CPU spent serializing and shipping logs
- Signal preservation: You still get representative traffic patterns and catch all critical issues
How It Works
Sampling decisions happen per request or per log statement. Common strategies:
- Random sampling: Log 1 in 100 requests at debug level
- Rate-based: Allow N debug logs per second
- Hash-based: Sample based on request/user ID for consistency
- Adaptive: Increase sample rate when error rates rise
Crucially, error and warning logs bypass sampling—you want 100% visibility into failures.
Example Scenario
Your payment service processes 50,000 requests/second. Full debug logging = 4.3 billion logs/day. With 1% sampling, you get 43 million representative samples, but every failed payment is still logged.
Key Takeaway: Sampling high-volume debug logs reduces costs exponentially while preserving complete visibility into errors, giving you the best of both worlds.