Retry Budgets and Rate Limiting
What you'll learn: How to use retry budgets to prevent retry storms from overwhelming your system when things go wrong.
The Problem with Unlimited Retries
You've learned about retry strategies like exponential backoff and jitter, but there's a dangerous scenario they don't fully solve: what happens when a service is already struggling and hundreds of clients all start retrying at once? Even with backoff, you can create a retry storm that prevents the struggling service from recovering.
What is a Retry Budget?
A retry budget is a cap on the percentage of requests that can be retried within a time window. Think of it like a safety valve on a pressure cooker—once you hit the limit, additional failures simply fail fast instead of adding more load.
For example, you might set a retry budget of 10%. If your service normally handles 1,000 requests per minute, you'll allow a maximum of 100 retries per minute. Once that budget is exhausted, clients stop retrying and return errors immediately.
How It Works
The system tracks:
- Total requests attempted in the current window
- Retry requests made in that same window
- Budget ratio = retries / total requests
When retries / total_requests exceeds your threshold (say, 10%), new retry attempts are blocked until the window resets or the ratio drops.
Why This Matters
During cascading failures, retry budgets prevent the "pile-on effect" where well-intentioned retries actually keep the failing service down. By limiting retry pressure, you give the struggling service breathing room to recover while still allowing some retries for transient glitches.
This pairs naturally with rate limiting—your rate limiter protects against external overload, while retry budgets protect against self-inflicted retry overload.
Key Takeaway:
Retry budgets cap the percentage of requests that can be retried, preventing cascading failures where retry storms overwhelm already-struggling services.