Percentiles and Aggregation
What you'll learn: Why percentiles like p50, p95, and p99 reveal the true user experience that averages conceal.
The Problem with Averages
When measuring response times, averages lie. Imagine a web service where 95% of requests complete in 100ms, but 5% take 5 seconds. The average might be around 350ms—which sounds fine—but 1 in 20 users is waiting five full seconds, experiencing a broken service.
Averages smooth out the pain. They give you a single number that hides the worst experiences your users are having.
Enter Percentiles
Percentiles slice your data distribution to show what percentage of requests fall below a certain threshold:
- p50 (median): Half of all requests are faster than this value
- p95: 95% of requests are faster; only 5% are slower
- p99: 99% are faster; this captures the "worst" 1% of experiences
If your p50 is 100ms but your p99 is 5 seconds, you know that most users are fine, but your worst-off users are suffering badly.
Why This Matters
Real users don't experience "average latency"—they experience their request. Percentiles tell you:
- Who's hurting: High p95/p99 values reveal outliers that averages mask
- Where capacity issues hide: Slow tail latencies often indicate resource contention, retries, or downstream failures
- What SLAs should target: Service level objectives often use p95 or p99 ("99% of requests under 200ms")
Analogy: If the average depth of a river is 3 feet, that's safe—unless one section is 10 feet deep. You need to know the distribution, not just the mean.
Key Takeaway: Always monitor p95 and p99 percentiles alongside averages; they reveal the critical latency outliers that destroy user experience even when your average looks healthy.