Batch Size and Throughput
What you'll learn: How to balance batch size to maximize inference speed without overwhelming your system.
When running inference with pre-trained models, you rarely process just one input at a time. Batch size is the number of inputs you process simultaneously in a single forward pass through the model. Throughput measures how many inputs you can process per second.
The Core Relationship
Think of batch size like loading a dishwasher. Processing one plate at a time wastes time and energy. But cramming too many plates in might not fit, or the machine might struggle. The sweet spot maximizes dishes cleaned per hour.
Small batch sizes (like 1-4) mean frequent model calls with underutilized hardware—especially GPUs, which excel at parallel computation. Your throughput suffers because you're not taking advantage of the hardware's full capacity.
Large batch sizes (like 64+) pack more work into each pass, dramatically improving throughput. However, memory requirements grow with batch size. Too large, and you'll hit out-of-memory errors. Even if it fits, extremely large batches may introduce latency—each individual request waits longer for the batch to fill and complete.
Finding Your Optimal Batch Size
Start small and increase the batch size while monitoring:
- Memory usage: Watch for OOM (out of memory) errors
- Throughput: Measure inputs processed per second
- Latency: Track time from request to response
Your optimal batch size sits just below your memory limit where throughput plateaus. This varies by model size, hardware, and input length.
Key Takeaway: Batch size directly impacts throughput—larger batches improve efficiency by utilizing hardware better, but you're constrained by available memory and acceptable latency for your use case.