Batch Size Selection
You'll learn: How to choose the right batch size by balancing memory constraints, throughput needs, and latency requirements in production LLM serving.
The Core Trade-Off
Batch size is the number of requests processed together in a single inference pass. Think of it like a restaurant kitchen deciding whether to cook meals one at a time or prepare multiple orders simultaneously. Larger batches use shared resources more efficiently, but each customer waits longer for their specific order.
The Four Dimensions
Memory Usage: Each request in a batch needs GPU memory for its input tokens, KV cache, and intermediate activations. Double the batch size, and you roughly double memory consumption. Run too large, and you'll hit out-of-memory errors.
Throughput: Larger batches maximize GPU utilization by amortizing overhead across many requests. A batch of 32 might process 10x more tokens per second than batch-size-1, because the GPU isn't sitting idle between requests.
Latency: Bigger batches mean each request waits for others to complete. If you're processing a batch of 64 requests and one has 2000 output tokens, all 64 responses are delayed. For real-time applications, this is unacceptable.
Hardware Utilization: Small batches waste GPU compute capacity. Large batches risk timeouts and poor user experience. The sweet spot depends on your specific GPU, model size, and SLA requirements.
Finding Your Number
Start with your memory ceiling: profile one request's memory, then increase batch size until you approach 80-90% GPU memory usage. Then test latency with realistic workloads. If P95 latency exceeds your target, reduce batch size. Monitor throughput metrics—if requests-per-second plateaus before hitting memory limits, you've found optimal density.
Key Takeaway: Optimal batch size maximizes throughput within your memory budget while meeting latency SLAs—test systematically rather than guessing, because the right number varies by model, hardware, and traffic patterns.