Throughput vs Latency Trade-offs
You'll learn how continuous batching maximizes throughput while managing per-request latency constraints in LLM inference engines.
The Fundamental Tension
Continuous batching creates a classic engineering trade-off: you want to maximize throughput (total requests processed per second) by keeping your GPU fully utilized, but you also need to minimize latency (time each individual request waits and executes). These goals often conflict.
How Continuous Batching Navigates This Trade-off
Think of continuous batching like a rideshare service. To maximize efficiency (throughput), you'd pick up as many passengers as possible before starting the route. But each new pickup makes existing passengers wait longer (latency).
In continuous batching:
- Higher throughput comes from larger batch sizes and aggressive request admission—you keep the GPU busy processing many requests simultaneously
- Lower latency requires smaller batches and faster scheduling—you prioritize getting individual requests through quickly
The key innovation: continuous batching adjusts batch composition dynamically at each iteration. When a request completes its generation, that slot immediately becomes available for a new request, rather than forcing the new request to wait for an entirely new batch.
Configuration Levers
You control this trade-off through:
- Maximum batch size: Caps throughput to protect latency
- Admission policies: How aggressively you accept new requests
- Preemption rules: Whether you pause long-running requests to serve urgent ones
- KV cache memory limits: Constrains how many concurrent requests you can handle
Real-World Balance
Production systems typically set latency SLOs (e.g., P95 < 500ms) and then maximize throughput within that constraint. Continuous batching excels here because it maintains high throughput without the dramatic latency spikes of static batching.
Key Takeaway: Continuous batching achieves better throughput-latency balance than static batching by dynamically adjusting batch composition, allowing you to maximize GPU utilization while respecting per-request latency requirements.