Latency vs Throughput
You'll learn: The crucial distinction between how fast a single request completes (latency) versus how many requests your system handles overall (throughput).
The Core Difference
Latency measures the time it takes to complete a single operation—from request initiation to response delivery. Think of it as "How long does one customer wait in line?"
Throughput measures how many operations your system completes per unit of time. Think of it as "How many customers does the store serve per hour?"
Why the Distinction Matters
These metrics often pull in opposite directions. You might optimize for low latency by processing requests immediately one-at-a-time, but this could limit throughput. Conversely, batching requests increases throughput but adds latency for individual requests waiting in the batch.
The Restaurant Analogy
Imagine a restaurant:
- Low latency approach: Each customer gets their meal prepared immediately, start to finish. Great experience for that customer (fast!), but the kitchen serves fewer total meals per hour.
- High throughput approach: The kitchen batches orders, cooking multiple steaks simultaneously. More customers fed per hour, but each individual waits longer.
Measuring Both
When you benchmark (as you learned in lesson 1200-1203), always measure both:
- Latency: Track response time percentiles (p50, p95, p99)
- Throughput: Measure requests per second your system sustains
Neither metric alone tells the complete performance story. A system with great average latency but low throughput won't handle real-world load. Similarly, high throughput means nothing if every request takes minutes to complete.
Key Takeaway: Latency tells you about user experience for individual requests; throughput tells you about system capacity—optimize for both based on your requirements, understanding the trade-offs between them.