Static vs Dynamic Batching
What You'll Learn
You'll understand how static and dynamic batching approaches differ when serving ML models, and why dynamic batching often wins for throughput and latency optimization.
The Core Idea
When serving models in production, batching groups multiple inference requests together to process them simultaneously, leveraging GPU parallelism. The key difference lies in when and how these batches are formed.
Static batching waits until a fixed number of requests accumulate (say, exactly 8 or 16) before processing them together. It's predictable but inefficient—if only 7 requests arrive, the system either waits indefinitely or processes a partial batch, wasting GPU capacity or adding latency.
Dynamic batching continuously monitors incoming requests and forms batches on-the-fly within a small time window (e.g., 10-50ms). If 3 requests arrive, process those 3. If 12 arrive in the next window, process 12. The batch size adapts to traffic patterns in real-time.
The Restaurant Analogy
Static batching is like a restaurant that seats customers only when exactly 4 people arrive—parties of 3 wait indefinitely, and tables sit empty.
Dynamic batching is like seating customers as they arrive, filling tables efficiently: sometimes 2 people, sometimes 6, maximizing both speed and capacity.
Why Dynamic Wins
Dynamic batching dramatically improves:
- Throughput: Better GPU utilization across varying traffic
- Latency: Requests don't wait for artificial batch size thresholds
- Resource efficiency: No idle GPU time or forced partial batches
Most modern serving frameworks (vLLM, TGI, Triton) implement dynamic batching by default, with configurable max batch sizes and timeout windows to balance throughput and latency.
Key Takeaway:
Dynamic batching adapts batch sizes to real-time request patterns, delivering better throughput and lower latency than static batching's fixed-size approach—essential for production LLM serving.