Thundering Herd After Recovery
What You'll Learn
You'll understand why a recovering service can be instantly crushed by a flood of simultaneous reconnection attempts from clients that have been waiting.
The Post-Outage Stampede
When a service fails, hundreds or thousands of clients may back off, retry with exponential delays, or wait with circuit breakers open. But when that service finally recovers and starts accepting requests again, all those waiting clients detect recovery at roughly the same time.
The result? A synchronized stampede of reconnection attempts that can immediately overwhelm the recovering service, causing it to fail again—starting a vicious cycle.
Why This Happens
Imagine a popular restaurant that closes for repairs. Word spreads when it reopens. Everyone who's been waiting shows up at the exact same moment, overwhelming the staff and kitchen before they've warmed up.
In distributed systems:
- Circuit breakers in the half-open state all test recovery simultaneously
- Queued requests flush all at once when backpressure clears
- Retry timers across many clients expire at similar times
- Load balancers detect health checks passing and route full traffic instantly
The recovering service, possibly still warming caches or scaling up resources, gets hit with 100% load before it's ready—and crashes again.
Prevention Through Gradual Recovery
Solutions include:
- Jittered reconnection delays so clients don't all retry at once
- Gradual circuit breaker resets that slowly increase allowed traffic
- Load shedding at the recovering service (accept only partial load initially)
- Progressive health checks that signal "ready but limited capacity"
- Client-side rate limiting during recovery detection
The goal is to allow the service to warm up gradually rather than face instant full load.
Key Takeaway: A service recovering from failure needs protection from its own clients—without staggered, rate-limited reconnections, successful recovery can immediately trigger another failure through synchronized stampedes.