Course contentsShow
← Software Engineering
Lesson 1204 of 1,93336. Performance EngineeringFree lesson

Latency vs Throughput

Understanding the difference between latency (response time) and throughput (requests per second) and why both matter.

Latency vs Throughput

You'll learn: The crucial distinction between how fast a single request completes (latency) versus how many requests your system handles overall (throughput).

The Core Difference

Latency measures the time it takes to complete a single operation—from request initiation to response delivery. Think of it as "How long does one customer wait in line?"

Throughput measures how many operations your system completes per unit of time. Think of it as "How many customers does the store serve per hour?"

Why the Distinction Matters

These metrics often pull in opposite directions. You might optimize for low latency by processing requests immediately one-at-a-time, but this could limit throughput. Conversely, batching requests increases throughput but adds latency for individual requests waiting in the batch.

The Restaurant Analogy

Imagine a restaurant:

  • Low latency approach: Each customer gets their meal prepared immediately, start to finish. Great experience for that customer (fast!), but the kitchen serves fewer total meals per hour.
  • High throughput approach: The kitchen batches orders, cooking multiple steaks simultaneously. More customers fed per hour, but each individual waits longer.

Measuring Both

When you benchmark (as you learned in lesson 1200-1203), always measure both:

  • Latency: Track response time percentiles (p50, p95, p99)
  • Throughput: Measure requests per second your system sustains

Neither metric alone tells the complete performance story. A system with great average latency but low throughput won't handle real-world load. Similarly, high throughput means nothing if every request takes minutes to complete.

Key Takeaway: Latency tells you about user experience for individual requests; throughput tells you about system capacity—optimize for both based on your requirements, understanding the trade-offs between them.