Course contentsShow
← System Design
Lesson 1717 of 1,91941. Real-World System: Notification SystemFree lesson

Handling Provider Throttling and Rate Limits

Designing retry logic and backpressure handling when external providers enforce rate limits.

Handling Provider Throttling and Rate Limits

What You'll Learn

You'll learn how to design retry logic and backpressure handling when external notification providers (SMS, email, push) enforce rate limits on your system.

The Challenge

External providers like Twilio (SMS), SendGrid (email), or FCM (push) impose rate limits—perhaps 1000 messages per second or 10,000 per hour. If your notification system tries to send faster than allowed, requests get rejected with errors like 429 Too Many Requests. Naively retrying makes things worse.

Core Design Patterns

Backpressure Handling

When a provider signals throttling, slow down upstream rather than piling up retries. Your worker pulls fewer messages from the queue, creating natural backpressure that prevents queue overflow.

Analogy: Like a water pipe with a narrow valve at the end—if you keep pumping water in, pressure builds and the pipe bursts. Instead, detect the restriction and reduce the flow rate.

Adaptive Retry Logic

Implement exponential backoff with jitter when rate-limited:

  • First retry: wait 1-2 seconds
  • Second retry: wait 2-4 seconds
  • Third retry: wait 4-8 seconds

Add randomness (jitter) so multiple workers don't retry simultaneously, creating synchronized thundering herds.

Token Bucket or Leaky Bucket

Maintain a local rate limiter that mirrors provider limits. Before sending, check if you have capacity. This prevents even making requests that will fail, reducing wasted API calls and improving efficiency.

Circuit Breaker Pattern

If a provider consistently returns throttling errors, open the circuit: temporarily stop sending to that provider, giving it time to recover. Route urgent notifications through fallback channels or queue them with longer delays.

Key Takeaway:
Handle provider throttling by implementing backpressure (slow down upstream processing), exponential backoff with jitter (spread out retries), local rate limiting (prevent overload), and circuit breakers (detect systemic issues early).