Async Log Shipping Architectures

In this lesson, you will learn how to design high-volume telemetry exporters that batch log data for efficient, asynchronous transmission to remote sinks.

The Bottleneck of Naive Logging

When building distributed systems, logging every event as an individual network request is a recipe for disaster. The overhead of repeatedly establishing connections and serializing small packets will saturate your network stack and introduce significant latency into your application’s critical path. Instead, we use an Async Log Shipping Architecture, where log events are pushed to an in-memory buffer, and a background worker drains that buffer in batches.

Implementing Batch Exporters

The goal is to decouple the production of logs from the consumption by the network layer. We achieve this using a tokio::sync::mpsc channel as a high-speed buffer. The producer sends log events into the channel, while a dedicated async worker task consumes them.

use tokio::sync::mpsc;
use std::time::Duration;

async fn telemetry_worker(mut rx: mpsc::Receiver<LogEvent>) {
    let mut batch = Vec::with_capacity(100);
    let mut interval = tokio::time::interval(Duration::from_millis(500));

    loop {
        tokio::select! {
            Some(log) = rx.recv() => {
                batch.push(log);
                if batch.len() >= 100 {
                    flush_batch(&mut batch).await;
                }
            }
            _ = interval.tick() => {
                if !batch.is_empty() {
                    flush_batch(&mut batch).await;
                }
            }
        }
    }
}

Why Batching Matters

By using a tokio::select! block, we implement a "trigger-or-timeout" strategy. The exporter flushes logs either when the buffer reaches a specific size (e.g., 100 logs) or when a time interval (e.g., 500ms) elapses. This ensures that:

1. Throughput remains high: You reduce the total number of network system calls.
2. Latency remains predictable: Even during low-traffic periods, logs are shipped within a maximum window defined by the timeout.
3. Backpressure: By setting a bound on your mpsc channel, you provide backpressure to the application, preventing it from overwhelming memory if the remote log collector slows down.

Key Takeaway: Efficient log shipping relies on decoupling event creation from network transmission via a batched, buffered worker that balances throughput against data freshness.