Course contentsShow
← System Design
Lesson 1188 of 1,91930. Observability: MetricsFree lesson

Percentiles and Tail Latencies

Understanding p50, p95, p99 percentiles and why tail latencies matter more than averages at scale.

Percentiles and Tail Latencies

What you'll learn: Why tracking p50, p95, and p99 percentiles reveals performance problems that averages hide—especially at scale.

The Problem with Averages

Imagine 99 customers experience 100ms response time, but one unlucky customer waits 10 seconds. The average is 199ms—not terrible. But that single customer had a miserable experience, and averages completely obscured it.

At scale, this matters enormously. If you serve millions of requests daily and 1% experience terrible latency, that's tens of thousands of frustrated users—yet your average latency looks fine.

What Are Percentiles?

Percentiles show latency at specific thresholds:

  • p50 (median): 50% of requests are faster than this value
  • p95: 95% of requests are faster than this (5% are slower)
  • p99: 99% of requests are faster than this (1% are slower)

If your p99 is 2 seconds, then 1 in 100 requests takes 2+ seconds. That's your tail latency—the slowest requests in the distribution.

Why Tail Latencies Matter

In distributed systems, tail latencies compound. If a user request fans out to 10 backend services, and each has a p99 of 100ms, the probability that at least one service hits that tail increases dramatically. The user's overall experience is dominated by the slowest component.

At scale, someone always hits the tail. Optimizing p99 and p999 directly improves the worst experiences for real users, not just theoretical metrics.

Key Takeaway: Percentiles (p95, p99) expose tail latencies that averages hide; at scale, even rare slow requests affect thousands of users and compound across service chains, making tail latency optimization critical for user experience.