Static vs Dynamic Batching
You'll learn: How static batching creates artificial delays by forcing all requests in a batch to wait for the slowest one to finish.
The Static Batching Problem
In static batching, the inference server collects requests until it reaches a predetermined batch size (or timeout), processes them together, and must wait for all requests in that batch to complete before starting the next batch.
Think of it like a bus that picks up passengers at scheduled times. Even if the bus fills up early, it won't leave until the scheduled departure. Worse, once everyone boards, the bus can't pick up new passengers until it completes the entire route and returns—even if it passes empty stops along the way.
The Core Limitation
The fundamental issue: all requests are blocked by the slowest request in the batch.
If you batch 8 inference requests together and 7 complete quickly but 1 takes 10x longer (maybe it's a longer sequence in an LLM), those 7 fast requests sit idle, waiting. Meanwhile, new requests queue up outside, unable to start even though GPU resources are becoming available.
This creates a cascading effect:
- Increased tail latency (P95, P99 metrics suffer)
- Underutilized GPU during the "waiting phase"
- Poor throughput when request durations vary significantly
- Artificial serialization of independent work
Why This Matters for LLMs
LLM inference is particularly affected because token generation is auto-regressive—different prompts generate different numbers of tokens. A short question might finish in 20 tokens while a complex one generates 500. Static batching forces them to lock-step together.
Key Takeaway: Static batching's requirement that all requests finish before new ones begin creates artificial bottlenecks, especially when request processing times vary—a critical limitation that dynamic batching and continuous batching techniques address.