Course contentsShow
← System Design
Lesson 1093 of 1,91928. Timeouts and DeadlinesFree lesson

The P99 Problem with Timeouts

Why timeout values must account for tail latencies and how P99/P999 metrics inform timeout choices.

The P99 Problem with Timeouts

You'll learn why setting timeouts based on average response times can still leave most requests hanging.

The Core Problem

When you set a timeout value, your instinct might be to look at your average response time — say 50ms — and set a timeout at 200ms, thinking "that's 4× the average, plenty of room!" But here's the trap: averages hide the worst cases.

In distributed systems, most requests complete quickly, but a small percentage take much longer due to:

  • Garbage collection pauses
  • Network retransmissions
  • Occasional slow disk reads
  • Cache misses or lock contention

If your P99 latency (99th percentile) is 800ms, but your timeout is 200ms, 1% of all requests will always time out — even when nothing is actually broken.

Why Tail Latencies Matter

The "P99 problem" means your timeout must be informed by tail latencies, not averages:

  • P50 (median): Half your requests complete faster
  • P99: 99% complete faster — but 1 in 100 naturally takes longer
  • P999: 99.9% complete faster

If your service handles 1000 requests/second and P99 is 800ms, that's 10 requests/second hitting the slow path under normal conditions. A 200ms timeout would kill these healthy-but-slow requests unnecessarily.

Setting Realistic Timeouts

Look at your P99 or P999 metrics under normal load. Your timeout should be slightly above these values to allow legitimate slow requests to complete, while still catching truly stuck operations.

Example: If P99 = 600ms, consider a timeout of 800-1000ms, not 200ms.

Key Takeaway: Timeout values must be based on tail latencies (P99/P999), not averages, to avoid unnecessarily killing slow-but-healthy requests while still protecting against truly stuck operations.