Course contentsShow
← System Design
Lesson 1164 of 1,91929. Observability: LoggingFree lesson

Sampling for High-Volume Logs

Sample verbose logs (like per-request debug info) to reduce cost while preserving error visibility.

Sampling for High-Volume Logs

You'll learn how to drastically cut logging costs and storage by selectively recording only a fraction of high-volume debug logs while keeping all errors visible.

The Core Idea

When a service handles millions of requests per second, logging every single request at DEBUG or INFO level becomes prohibitively expensive—both in storage costs and ingestion overhead. Sampling means intentionally recording only a subset of these verbose logs (say, 1% or 0.1%), while still logging all errors and warnings at full rate.

Think of it like quality control in manufacturing: you don't inspect every single widget coming off the assembly line—you sample periodically to monitor normal operation. But if a widget breaks, you always log that failure.

Why Sample?

  • Cost reduction: Storing 1% of debug logs instead of 100% can reduce your logging bill by 99%
  • Performance: Less I/O and CPU spent serializing and shipping logs
  • Signal preservation: You still get representative traffic patterns and catch all critical issues

How It Works

Sampling decisions happen per request or per log statement. Common strategies:

  • Random sampling: Log 1 in 100 requests at debug level
  • Rate-based: Allow N debug logs per second
  • Hash-based: Sample based on request/user ID for consistency
  • Adaptive: Increase sample rate when error rates rise

Crucially, error and warning logs bypass sampling—you want 100% visibility into failures.

Example Scenario

Your payment service processes 50,000 requests/second. Full debug logging = 4.3 billion logs/day. With 1% sampling, you get 43 million representative samples, but every failed payment is still logged.

Key Takeaway: Sampling high-volume debug logs reduces costs exponentially while preserving complete visibility into errors, giving you the best of both worlds.