Batch Size Selection for Inference
You'll learn: How to choose the right batch size that maximizes GPU efficiency while respecting memory limits and latency constraints during model inference.
The Core Challenge
Selecting batch size for inference is like packing a delivery truck: pack too few items (small batch) and you waste fuel and capacity; pack too many (large batch) and you might exceed weight limits or delay urgent deliveries. You need the sweet spot that balances three competing goals:
- GPU Utilization: Larger batches saturate compute units better, increasing throughput
- Memory Constraints: Batch size × model size × intermediate activations must fit in GPU memory
- Latency Requirements: Larger batches mean waiting longer to accumulate requests
Finding the Optimal Batch Size
Start by measuring memory headroom. Your maximum batch size is bounded by:
Available GPU Memory ≥ Model Weights + (Batch Size × Per-Sample Memory)
Next, profile throughput at different batch sizes. You'll typically see diminishing returns: doubling from 1→2 might give 1.8× throughput, but 64→128 might only yield 1.1×.
Finally, consider your latency budget. If you must respond within 100ms and batching adds 20ms wait time, you have 80ms left for actual inference. Dynamic batching helps here—accumulate requests up to a timeout, then process whatever you have.
The Practical Approach
Test empirically: measure latency percentiles (p50, p95, p99) and throughput at batch sizes like 1, 2, 4, 8, 16, 32, 64. The "knee" in the throughput curve—where gains flatten—often indicates your optimal choice. For real-time APIs, smaller batches (1-8) prioritize latency; for batch jobs, maximize (32-256+) for throughput.
Key Takeaway: Optimal inference batch size sits at the intersection of maximum GPU memory utilization, acceptable latency thresholds, and diminishing throughput returns—always profile your specific model and hardware combination.