Course contentsShow
← System Design
Lesson 1117 of 1,91928. Timeouts and DeadlinesFree lesson

Adaptive Timeouts Based on Historical Latency

Dynamically adjust timeout values using percentile-based calculations from recent request latencies.

Adaptive Timeouts Based on Historical Latency

You'll learn how to dynamically adjust timeout values using recent request performance data instead of hardcoding them.

The Problem with Static Timeouts

Fixed timeouts (say, always 5 seconds) ignore reality: your dependency might normally respond in 100ms but occasionally take 3 seconds during load spikes. A static timeout either fails prematurely during normal variance or waits too long during actual failures.

The Core Idea

Adaptive timeouts continuously measure actual request latencies and adjust timeout values based on percentile calculations—typically P95 or P99. If your service's P99 latency over the last 5 minutes is 800ms, you might set your timeout to P99 + buffer, like 1200ms. As latency patterns shift, your timeout shifts with them.

How It Works

  1. Track recent latencies in a sliding time window (e.g., last 1000 requests or last 5 minutes)
  2. Calculate percentiles from this distribution (P50, P95, P99)
  3. Set timeout = P(chosen percentile) × multiplier (e.g., P99 × 1.5)
  4. Refresh periodically as new data arrives and old data ages out

The Analogy

Think of a traffic route planner that learns: instead of always budgeting 30 minutes regardless of traffic, it checks recent drive times and says "lately P95 is 42 minutes, so we'll budget 50 minutes to be safe."

Benefits and Tradeoffs

Pros: Timeouts stay relevant as system behavior changes; avoids premature failures during acceptable slowdowns.

Cons: Requires latency tracking infrastructure; can be slow to react to sudden degradation; complex to tune the percentile/multiplier choice.

Key Takeaway:
Adaptive timeouts use percentile-based calculations from recent request latencies to dynamically adjust timeout values, keeping them aligned with actual system behavior rather than static guesses.