Latency vs Throughput Tradeoffs
You'll learn: Why batch and stream processing make opposite choices when balancing speed-per-record against total volume processed.
The Fundamental Tension
In data processing, latency measures how quickly a single record moves from input to output (time-to-result), while throughput measures total records processed per unit time (volume-per-second). These two goals often conflict—optimizing one typically sacrifices the other.
Batch processing maximizes throughput by collecting data into large groups before processing. Like loading a truck completely full before driving to the warehouse, you amortize overhead costs (startup time, I/O operations, network round-trips) across thousands or millions of records. Each individual record waits longer (high latency), but you process vastly more records per hour (high throughput).
Stream processing minimizes latency by handling records immediately as they arrive. Like delivering packages one at a time by bicycle, each package arrives quickly (low latency), but your total delivery capacity is limited (lower throughput). You pay overhead costs repeatedly—once per record or small micro-batch.
Why the Tradeoff Exists
Processing systems have fixed costs: opening database connections, allocating memory, coordinating workers, serializing data. Batch amortizes these costs across huge datasets. Streaming pays them frequently for responsiveness.
Think of cooking: batch is preparing 100 meals at once in a commercial kitchen (efficient, but dinner's late), while streaming is cooking each order immediately at a diner (fast service, but the kitchen handles fewer total orders per hour).
Your choice depends on your use case: financial fraud detection needs streaming's low latency, while monthly sales reports benefit from batch's high throughput.
Key Takeaway:
Batch trades latency for throughput (process more, wait longer); streaming trades throughput for latency (process less, respond faster).