Latency vs Throughput: The Fundamental Tradeoff
You'll learn: Why optimizing for fast individual predictions often conflicts with maximizing total system capacity, and how to choose the right balance.
The Core Tension
Latency measures how long a single inference request takes from start to finish—the time one user waits for a prediction. Throughput measures how many requests your system handles per second—your total capacity.
These metrics have an inverse relationship: optimizations that reduce latency often decrease throughput, and vice versa.
Why They Conflict
Think of a restaurant: serving each customer immediately (low latency) means chefs prepare one meal at a time. But batch-cooking multiple orders together (high throughput) makes each individual customer wait longer.
In ML serving:
- Low latency: Process requests individually, minimal batching, no queuing → fewer requests/second
- High throughput: Use dynamic batching, larger batch sizes, accept queuing delays → slower individual responses
When you configured batch sizes and timeouts in earlier lessons, you were navigating this exact tradeoff.
When to Optimize for Each
Optimize for latency when:
- Real-time user interactions (chatbots, recommendation systems)
- Strict SLAs (e.g., "predictions under 100ms")
- User experience depends on responsiveness
Optimize for throughput when:
- Batch processing pipelines
- Cost efficiency matters more than individual speed
- Background tasks without user waiting
- High-volume, non-interactive workloads
The Sweet Spot
Most production systems need balance. You might accept 50ms extra latency (from batching) to triple your throughput and cut infrastructure costs by 60%. This is why batch timeout configuration matters—it's your knob for adjusting this tradeoff.
Key Takeaway: Latency and throughput are inversely related; optimizing one typically sacrifices the other. Choose based on whether individual response time or total system capacity matters more for your use case.