Batching Trade-offs: Latency vs Throughput
What you'll learn: How batching inference requests creates a fundamental tension between minimizing response time for individual requests and maximizing overall system capacity.
The Core Trade-off
When serving ML models, batching multiple inference requests together improves throughput (requests processed per second) by leveraging GPU parallelism and reducing overhead. However, this comes at the cost of increased latency (time for a single request to complete), because early-arriving requests must wait for the batch to fill before processing begins.
Think of it like a shuttle bus service: waiting for more passengers before departing increases vehicle efficiency (throughput) but makes early arrivals wait longer (latency).
Understanding the Balance
Without batching:
- Each request processes immediately (low latency)
- GPU underutilized, processing one sample at a time (low throughput)
- High per-request overhead
With aggressive batching:
- Requests wait for batch to fill (high latency)
- GPU runs at full capacity (high throughput)
- Minimal per-request overhead
The Sweet Spot
The optimal batch size depends on your use case:
- Latency-sensitive applications (chatbots, real-time recommendations): smaller batches, timeout-based triggering
- Throughput-focused workloads (batch processing, offline inference): larger batches, maximize GPU utilization
- Balanced scenarios: dynamic batching with max wait time limits (as covered in the previous lesson)
You must measure both metrics simultaneously—improving one often degrades the other. The "right" configuration balances business requirements (acceptable latency SLA) against infrastructure costs (servers needed to handle load).
Key Takeaway: Batching inference requests forces you to choose between fast individual responses (low latency) and high system capacity (throughput)—tune batch size and timeout parameters based on your application's tolerance for wait time versus its need to handle request volume efficiently.