Handling Provider Throttling and Rate Limits
What You'll Learn
You'll learn how to design retry logic and backpressure handling when external notification providers (SMS, email, push) enforce rate limits on your system.
The Challenge
External providers like Twilio (SMS), SendGrid (email), or FCM (push) impose rate limits—perhaps 1000 messages per second or 10,000 per hour. If your notification system tries to send faster than allowed, requests get rejected with errors like 429 Too Many Requests. Naively retrying makes things worse.
Core Design Patterns
Backpressure Handling
When a provider signals throttling, slow down upstream rather than piling up retries. Your worker pulls fewer messages from the queue, creating natural backpressure that prevents queue overflow.
Analogy: Like a water pipe with a narrow valve at the end—if you keep pumping water in, pressure builds and the pipe bursts. Instead, detect the restriction and reduce the flow rate.
Adaptive Retry Logic
Implement exponential backoff with jitter when rate-limited:
- First retry: wait 1-2 seconds
- Second retry: wait 2-4 seconds
- Third retry: wait 4-8 seconds
Add randomness (jitter) so multiple workers don't retry simultaneously, creating synchronized thundering herds.
Token Bucket or Leaky Bucket
Maintain a local rate limiter that mirrors provider limits. Before sending, check if you have capacity. This prevents even making requests that will fail, reducing wasted API calls and improving efficiency.
Circuit Breaker Pattern
If a provider consistently returns throttling errors, open the circuit: temporarily stop sending to that provider, giving it time to recover. Route urgent notifications through fallback channels or queue them with longer delays.
Key Takeaway:
Handle provider throttling by implementing backpressure (slow down upstream processing), exponential backoff with jitter (spread out retries), local rate limiting (prevent overload), and circuit breakers (detect systemic issues early).