Batch Size Selection and Timeout Configuration
What You'll Learn
You'll learn how to determine the optimal batch size and timeout windows for your inference server by balancing traffic patterns, SLA requirements, and hardware capabilities.
The Core Challenge
When implementing dynamic batching (which you've already learned collects multiple requests into a single inference batch), you face two critical configuration decisions:
Batch Size: How many requests should we accumulate before processing? Timeout: How long should we wait for a batch to fill before processing it anyway?
These parameters directly control your latency-throughput trade-off.
Finding the Sweet Spot
Think of it like a rideshare service: You want to fill the car (maximize throughput), but passengers can't wait forever (meet latency SLAs).
Batch Size Selection
Your maximum batch size should be determined by:
- Hardware memory limits: GPU memory constrains how many samples fit simultaneously
- Model architecture: Transformer models scale differently than CNNs
- Throughput targets: Larger batches maximize GPU utilization
- SLA requirements: Bigger batches mean some requests wait longer
Start by testing your model's memory footprint: gradually increase batch size until you hit OOM errors, then back off 20-30% for safety.
Timeout Configuration
Your timeout window answers: "How long do we wait for a batch to fill?"
- Low traffic: Short timeouts prevent requests from waiting unnecessarily
- High traffic: Longer timeouts allow batches to fill completely
- P99 latency SLA: Timeout must be significantly less than your SLA budget
Example thinking: If your SLA is 100ms and inference takes 40ms, your maximum safe timeout is ~50ms (leaving margin for networking and postprocessing).
Adaptive Strategies
Advanced systems adjust parameters based on observed traffic patterns—smaller batches and shorter timeouts during quiet periods, larger configurations during peak load.
Key Takeaway: Optimal batch size balances hardware capacity with throughput goals, while timeout configuration ensures you meet latency SLAs even when traffic is insufficient to fill batches naturally. Test empirically under realistic load patterns.