Batch Size and Throughput Planning
What You'll Learn
You'll learn how to calculate the optimal batch size for your self-hosted LLM deployment to maximize requests processed per second while keeping response times acceptable.
The Core Concept
Batch size is the number of requests your serving infrastructure processes simultaneously. Think of it like a restaurant kitchen: processing orders one-by-one is inefficient, but trying to cook 100 meals at once overwhelms the staff and slows everything down. The sweet spot depends on your "kitchen size" (hardware).
Throughput measures requests completed per second, while latency tracks how long each request takes. These metrics trade off against each other based on batch size.
Planning Your Batch Size
Start with your hardware constraints:
- GPU memory limits total tokens in flight (input + output across all requests)
- Compute capacity determines how many parallel operations you can handle efficiently
Calculate available token budget:
Available Memory = GPU Memory - Model Weights - KV Cache Overhead
Max Batch Tokens = Available Memory / Memory per Token
Then consider your latency requirements. Larger batches increase throughput but add queuing delays. If users expect sub-second responses, you need aggressive batching with short timeouts. For async workflows, you can batch more aggressively.
The Planning Process
- Profile single-request performance to establish baseline latency
- Test incrementally larger batches (2, 4, 8, 16, 32...)
- Monitor both metrics: throughput should rise, latency should remain acceptable
- Find the knee of the curve: where throughput gains flatten but latency keeps climbing
- Build in headroom: use 70-80% of maximum to handle traffic spikes
Key Takeaway: Optimal batch size balances hardware utilization against latency requirements—test empirically with your specific model, hardware, and traffic patterns to find the sweet spot that maximizes throughput without violating SLAs.