Course contentsShow
← AI Engineering
Lesson 1017 of 1,88625. Model Serving and Inference OptimizationFree lesson

Static vs Dynamic Batching

Compare static batch processing with dynamic batching that groups requests on-the-fly for better throughput and latency.

Static vs Dynamic Batching

What You'll Learn

You'll understand how static and dynamic batching approaches differ when serving ML models, and why dynamic batching often wins for throughput and latency optimization.

The Core Idea

When serving models in production, batching groups multiple inference requests together to process them simultaneously, leveraging GPU parallelism. The key difference lies in when and how these batches are formed.

Static batching waits until a fixed number of requests accumulate (say, exactly 8 or 16) before processing them together. It's predictable but inefficient—if only 7 requests arrive, the system either waits indefinitely or processes a partial batch, wasting GPU capacity or adding latency.

Dynamic batching continuously monitors incoming requests and forms batches on-the-fly within a small time window (e.g., 10-50ms). If 3 requests arrive, process those 3. If 12 arrive in the next window, process 12. The batch size adapts to traffic patterns in real-time.

The Restaurant Analogy

Static batching is like a restaurant that seats customers only when exactly 4 people arrive—parties of 3 wait indefinitely, and tables sit empty.

Dynamic batching is like seating customers as they arrive, filling tables efficiently: sometimes 2 people, sometimes 6, maximizing both speed and capacity.

Why Dynamic Wins

Dynamic batching dramatically improves:

  • Throughput: Better GPU utilization across varying traffic
  • Latency: Requests don't wait for artificial batch size thresholds
  • Resource efficiency: No idle GPU time or forced partial batches

Most modern serving frameworks (vLLM, TGI, Triton) implement dynamic batching by default, with configurable max batch sizes and timeout windows to balance throughput and latency.

Key Takeaway:
Dynamic batching adapts batch sizes to real-time request patterns, delivering better throughput and lower latency than static batching's fixed-size approach—essential for production LLM serving.