Course contentsShow
← Machine Learning and Deep Learning
Lesson 2936 of 3,53864. GPU Inference OptimizationFree lesson

Batch Size Selection for Inference

Determine optimal batch sizes balancing GPU utilization, memory constraints, and latency requirements.

Batch Size Selection for Inference

You'll learn: How to choose the right batch size that maximizes GPU efficiency while respecting memory limits and latency constraints during model inference.

The Core Challenge

Selecting batch size for inference is like packing a delivery truck: pack too few items (small batch) and you waste fuel and capacity; pack too many (large batch) and you might exceed weight limits or delay urgent deliveries. You need the sweet spot that balances three competing goals:

  1. GPU Utilization: Larger batches saturate compute units better, increasing throughput
  2. Memory Constraints: Batch size × model size × intermediate activations must fit in GPU memory
  3. Latency Requirements: Larger batches mean waiting longer to accumulate requests

Finding the Optimal Batch Size

Start by measuring memory headroom. Your maximum batch size is bounded by:

Available GPU Memory ≥ Model Weights + (Batch Size × Per-Sample Memory)

Next, profile throughput at different batch sizes. You'll typically see diminishing returns: doubling from 1→2 might give 1.8× throughput, but 64→128 might only yield 1.1×.

Finally, consider your latency budget. If you must respond within 100ms and batching adds 20ms wait time, you have 80ms left for actual inference. Dynamic batching helps here—accumulate requests up to a timeout, then process whatever you have.

The Practical Approach

Test empirically: measure latency percentiles (p50, p95, p99) and throughput at batch sizes like 1, 2, 4, 8, 16, 32, 64. The "knee" in the throughput curve—where gains flatten—often indicates your optimal choice. For real-time APIs, smaller batches (1-8) prioritize latency; for batch jobs, maximize (32-256+) for throughput.

Key Takeaway: Optimal inference batch size sits at the intersection of maximum GPU memory utilization, acceptable latency thresholds, and diminishing throughput returns—always profile your specific model and hardware combination.