Batching for Throughput: Static vs Dynamic
You'll learn how static and dynamic batching strategies trade off latency for throughput differently, and when each makes sense.
The Core Tradeoff
Both static and dynamic batching increase throughput by processing multiple requests together, but they differ fundamentally in when they form batches and how they handle the latency-throughput balance.
Static Batching
Static batching groups a fixed number of requests before processing, regardless of wait time. It's like waiting for a bus to fill completely before departing—maximum efficiency per trip, but passengers who arrive early wait longer.
Characteristics:
- Fixed batch size (e.g., always 32 requests)
- Predictable throughput and resource utilization
- Higher average latency due to mandatory wait times
- Best for offline/batch workloads where latency isn't critical
When to use: High-volume scenarios where you're willing to accept higher tail latencies for maximum GPU/TPU utilization. Perfect for overnight batch scoring or periodic bulk inference.
Dynamic Batching
Dynamic batching (covered earlier) collects requests opportunistically within a time window. It's like a bus that leaves when full or after a maximum wait time—balancing passenger convenience with efficiency.
Characteristics:
- Variable batch sizes (1 to max_batch_size)
- Lower latency for early requests
- Adapts to traffic patterns automatically
- Slightly lower peak throughput than static
When to use: Online serving where user-facing latency matters. The timeout ensures no request waits indefinitely, while still capturing batching benefits during traffic spikes.
The Decision Matrix
Choose static batching when throughput maximization justifies adding seconds of latency. Choose dynamic batching when you need to balance both, with latency as a hard constraint.
Key Takeaway: Static batching maximizes throughput at the cost of predictable, often high latency by always waiting for full batches; use it for offline workloads. Dynamic batching adapts batch size to traffic, keeping latency bounded while capturing opportunistic throughput gains—ideal for online serving.