The Rate Limiting Pattern
What you'll learn: How to protect your systems from overload by controlling the rate at which requests are processed.
What Is Rate Limiting?
Rate limiting is a resilience pattern that restricts how many requests a system accepts within a given time window. Think of it like a nightclub bouncer who only lets a certain number of people in per minute—no matter how many are waiting outside. This prevents the venue (your system) from becoming dangerously overcrowded.
Without rate limiting, a sudden spike in traffic—whether from legitimate users, buggy clients, or malicious actors—can overwhelm your servers, degrading performance for everyone or causing complete outages.
Common Rate Limiting Algorithms
Token Bucket
Imagine a bucket that holds tokens. Each token represents permission to process one request. The bucket refills at a steady rate (say, 10 tokens per second) up to a maximum capacity. When a request arrives, it consumes a token. If no tokens are available, the request is rejected or delayed.
This allows brief bursts above the baseline rate (using accumulated tokens) while preventing sustained overload.
Leaky Bucket
Picture a bucket with a small hole at the bottom. Requests pour in at the top, but they leak out (get processed) at a constant, controlled rate. If the bucket overflows, excess requests are discarded.
This enforces smooth, predictable output even when input is bursty.
Where to Apply Rate Limiting
Rate limiting typically happens at API gateways, load balancers, or within application code. You might limit requests per user, per API key, per IP address, or globally across your entire system—depending on your failure modes and fairness requirements.
Key Takeaway: Rate limiting protects systems from overload by controlling request rates using algorithms like token bucket (allowing bursts) or leaky bucket (enforcing smooth flow), ensuring stability under high or unpredictable load.