Aggregating Logs for Distributed Systems

In this lesson, you will learn how to design log shipping patterns that offload telemetry processing from your application's hot path to ensure that logging never becomes a performance bottleneck.

The Problem: Blocking the Hot Path

In a distributed system, writing logs to a central server synchronously (e.g., waiting for an HTTP POST to finish) is a performance killer. If your logging infrastructure experiences latency, your application threads will block, causing request queues to swell and latency to spike. Even simple file I/O can cause contention if the disk becomes saturated.

The Solution: Asynchronous Buffering

To decouple application logic from log persistence, we must implement an asynchronous producer-consumer pattern. The application thread should only be responsible for handing the log entry to an in-memory, thread-safe queue. A background worker then drains this queue and performs the "heavy lifting"—batching, compressing, and shipping logs over the network.

When implementing this in C#, avoid allocating memory per log entry to keep GC pressure low. Use a Channel<T> to provide a high-performance, bounded queue:

// High-performance bounded channel for log buffering
private readonly Channel<LogEntry> _logChannel = Channel.CreateBounded<LogEntry>(
    new BoundedChannelOptions(10000) { FullMode = BoundedChannelFullMode.DropOldest });

public void Log(LogEntry entry)
{
    // Try to write without blocking the caller
    _logChannel.Writer.TryWrite(entry);
}

// Background process to ship logs in batches
private async Task ShipLogsAsync(CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        var batch = new List<LogEntry>();
        while (batch.Count < 500 && _logChannel.Reader.TryRead(out var entry))
        {
            batch.Add(entry);
        }
        
        if (batch.Count > 0)
        {
            await SendToCentralServerAsync(batch);
        }
    }
}

Critical Considerations

1. Backpressure Strategy: When the channel hits its capacity, you must decide whether to DropOldest (preserve system availability) or Wait (guarantee log delivery but risk blocking the application). In high-throughput distributed systems, DropOldest is usually preferred to prevent the logger from crashing the primary service.
2. Batching: Shipping individual logs is inefficient due to network overhead. Always aggregate logs into batches to reduce the number of RPC calls, which helps maintain low latency in your distributed tracing.

Key Takeaway: Decouple log generation from log shipping using bounded, non-blocking in-memory buffers to ensure your observability infrastructure never starves your primary execution threads.