Course contentsShow
← System Design
Lesson 12 of 1,9191. System Design Mindset and First PrinciplesFree lesson

Performance Requirements: Latency and Throughput

Define and distinguish latency (response time) and throughput (requests per second) as performance metrics.

Performance Requirements: Latency and Throughput

What you'll learn: How to measure a system's speed using two fundamental metrics: latency and throughput.

The Core Idea

When we evaluate how well a system performs, we need precise ways to measure "speed." Two essential metrics help us do this:

Latency is the time it takes for a single request to complete — from when a user clicks "submit" to when they see a response. Think of it as "response time." We typically measure this in milliseconds (ms) or seconds.

Throughput is how many requests the system can handle in a given time period — usually expressed as requests per second (RPS) or queries per second (QPS). It measures the system's overall capacity.

Understanding the Difference

Imagine a restaurant:

  • Latency is how long it takes for one customer to receive their meal after ordering (e.g., 15 minutes).
  • Throughput is how many customers the restaurant can serve per hour (e.g., 60 customers/hour).

A restaurant might serve you quickly (low latency) but only handle a few customers at once (low throughput). Or it might process many orders simultaneously (high throughput) but each takes a while (high latency).

Why Both Matter

These metrics often exist in tension. You can't optimize one without considering the other:

  • A system with excellent latency (50ms per request) might have poor throughput if it can only handle 10 requests per second.
  • A system with high throughput (10,000 requests per second) might frustrate users if each request takes 5 seconds to complete.

During system design, you'll identify specific targets: "We need 200ms latency for 95% of requests" or "The system must handle 50,000 requests per second."

Key Takeaway: Latency measures individual request speed (response time), while throughput measures system capacity (requests per second) — both are critical performance requirements that guide your design decisions.