Timeout Jitter to Prevent Thundering Herds
What you'll learn: How adding randomization to timeout values prevents all clients from retrying simultaneously after a failure.
The Thundering Herd Problem with Timeouts
When many clients set identical timeouts (say, 5 seconds) and a backend service becomes slow or unresponsive, all those clients will timeout at exactly the same moment. They'll then retry simultaneously, creating a massive spike of traffic—a thundering herd—that can overwhelm the recovering service and trigger another cascading failure.
This is especially dangerous after an outage: the moment the service starts recovering, it gets slammed with synchronized retry traffic from thousands of clients who all timed out at the same instant.
How Timeout Jitter Works
Instead of using a fixed timeout value, you add jitter—random variation—to spread retries over time:
base_timeout = 5000ms
jitter = random(0, 1000ms)
actual_timeout = base_timeout + jitter
// Results: 5000-6000ms, distributed randomly
Now clients timeout between 5-6 seconds instead of all at 5 seconds. This spreads the retry load over a full second window rather than a single millisecond spike.
The Traffic Smoothing Effect
Think of jitter like staggering traffic lights at intersections. Without it, every car arrives at the next intersection simultaneously. With jitter, arrivals spread out naturally, preventing congestion.
The randomization breaks synchronization between clients who started requests at similar times, turning a sharp retry spike into a gradual ramp-up that gives the backend breathing room to recover.
Key Takeaway: Adding random jitter to timeout values (±10-20% is typical) prevents synchronized retry storms by distributing client timeout moments across a time window, protecting recovering services from thundering herd traffic.