Throughput vs Latency
What you'll learn: You'll understand the crucial difference between how quickly a system responds to individual requests (latency) and how much total work it processes over time (throughput).
The Core Distinction
Imagine you're at a coffee shop. Latency is how long it takes from ordering your coffee until you receive it in your hand. Throughput is how many coffees the shop can serve per hour.
These two metrics measure different aspects of performance:
-
Latency answers: "How fast is each individual operation?"
- Measured in time units: milliseconds, seconds
- Example: Your API responds in 50ms
-
Throughput answers: "How much work gets done overall?"
- Measured in operations per time: requests/second, transactions/minute
- Example: Your system handles 10,000 requests per second
Why Both Matter
A system can have:
- Low latency but low throughput: Like a single skilled barista making perfect coffee quickly, but only serving one customer at a time
- High latency but high throughput: Like an automated factory that takes 5 minutes per coffee but produces 1,000 simultaneously
- The ideal: Low latency AND high throughput (though this usually costs more)
The Tradeoff
You often face tradeoffs. Adding more parallel processing might increase throughput (more work done) but could increase individual latency (each request waits in line longer). Optimizing for one doesn't automatically optimize for the other.
Key Takeaway: Latency measures individual response speed; throughput measures total work capacity. Both are critical performance metrics, but they describe different system characteristics and often require different optimization strategies.