Course contentsShow
← Software Engineering
Lesson 1188 of 1,93336. Performance EngineeringFree lesson

Latency vs Throughput Trade-offs

Understanding the difference between latency (response time) and throughput (requests per second) and their relationship.

Latency vs Throughput Trade-offs

You'll learn: How optimizing for fast response times often conflicts with maximizing request volume, and why you need to balance both.

The Core Tension

Latency measures how long a single request takes to complete (response time), while throughput measures how many requests your system can handle per second. These two metrics often pull in opposite directions.

Think of a coffee shop: latency is how long one customer waits for their drink. Throughput is how many drinks the shop makes per hour. You could serve each customer instantly by dedicating all baristas to one order at a time (low latency), but total drinks per hour would plummet. Or you could batch orders and pipeline work for maximum output (high throughput), but individual wait times would spike.

The Trade-off in Practice

Optimizing for low latency often means:

  • Processing requests immediately without batching
  • Keeping resources idle to respond instantly
  • Prioritizing quick turnaround over efficient resource use

Optimizing for high throughput often means:

  • Batching requests together for efficiency
  • Queuing requests to maximize CPU/network utilization
  • Adding parallel processing that increases individual response times

For example, a database can achieve higher throughput by grouping writes into batches, but this adds latency since writes must wait for the batch to fill. Conversely, flushing every write immediately gives low latency but wastes potential throughput from efficient batch operations.

Finding the Balance

Neither metric alone tells the full story. You need both:

  • A system with high throughput but 30-second latency is unusable for users
  • A system with 10ms latency but only 1 req/sec throughput can't scale

Your SLOs (lesson 1174) should define acceptable targets for both based on user needs and business requirements.

Key Takeaway: Latency and throughput often compete—optimizing one typically degrades the other. Design systems with explicit targets for both metrics rather than maximizing either in isolation.