Latency vs Throughput Trade-offs
You'll learn: How optimizing for fast response times often conflicts with maximizing request volume, and why you need to balance both.
The Core Tension
Latency measures how long a single request takes to complete (response time), while throughput measures how many requests your system can handle per second. These two metrics often pull in opposite directions.
Think of a coffee shop: latency is how long one customer waits for their drink. Throughput is how many drinks the shop makes per hour. You could serve each customer instantly by dedicating all baristas to one order at a time (low latency), but total drinks per hour would plummet. Or you could batch orders and pipeline work for maximum output (high throughput), but individual wait times would spike.
The Trade-off in Practice
Optimizing for low latency often means:
- Processing requests immediately without batching
- Keeping resources idle to respond instantly
- Prioritizing quick turnaround over efficient resource use
Optimizing for high throughput often means:
- Batching requests together for efficiency
- Queuing requests to maximize CPU/network utilization
- Adding parallel processing that increases individual response times
For example, a database can achieve higher throughput by grouping writes into batches, but this adds latency since writes must wait for the batch to fill. Conversely, flushing every write immediately gives low latency but wastes potential throughput from efficient batch operations.
Finding the Balance
Neither metric alone tells the full story. You need both:
- A system with high throughput but 30-second latency is unusable for users
- A system with 10ms latency but only 1 req/sec throughput can't scale
Your SLOs (lesson 1174) should define acceptable targets for both based on user needs and business requirements.
Key Takeaway: Latency and throughput often compete—optimizing one typically degrades the other. Design systems with explicit targets for both metrics rather than maximizing either in isolation.