Sampling Logs for High-Traffic Applications

In this lesson, you will learn how to implement probabilistic logging to maintain system observability in high-traffic applications while significantly reducing storage and ingestion costs.

As applications scale, logging every single request or event can quickly become a performance bottleneck and a financial burden. When you are processing thousands of requests per second, you rarely need a 100% trace of every success to diagnose systemic issues. Instead, we use Probabilistic Sampling to log only a representative subset of events.

Implementing Probabilistic Sampling

The core idea is to introduce a "roll of the dice" before the logging statement. By using a pseudo-random number generator, we decide whether to emit a log entry based on a configured sampling rate.

public class LogSampler
{
    private readonly double _samplingRate; // e.g., 0.1 for 10%
    private readonly Random _random = new();

    public LogSampler(double samplingRate) => _samplingRate = samplingRate;

    public bool ShouldLog() => _random.NextDouble() < _samplingRate;
}

// Usage in your service
if (_logSampler.ShouldLog())
{
    _logger.LogInformation("Processing high-traffic request: {RequestId}", requestId);
}

Why this works for High-Traffic Systems

When you apply this pattern, you effectively decouple your diagnostic overhead from your traffic volume. If your traffic spikes by 10x, your logging volume remains constant because the sampling logic discards 90% of the entries at the source. This prevents I/O saturation and log-aggregation pipeline backpressure, which are common killers of high-throughput services.

However, be careful with your sampling strategy. For debugging, you should always log 100% of errors regardless of the sampling rate. You can achieve this by combining your sampler with conditional logic:

if (isError || _logSampler.ShouldLog())
{
    _logger.LogError(exception, "Request failed");
}

By ensuring that "interesting" events (like exceptions) bypass the sampler, you maintain full visibility into critical failures while keeping the noise from routine successful requests under control. This balance is essential for maintaining a high-performance system that remains debuggable without breaking your storage budget.

Key Takeaway: Probabilistic logging allows you to maintain statistical observability by emitting a fraction of total traffic, provided you always bypass the sampler for critical errors.