Course contentsShow
← Cloud and Infrastructure
Lesson 2744 of 2,98959. Queueing and Async PatternsFree lesson

Rate Limiting Fundamentals

Understand rate limiting concepts, why it's needed to protect services, and common algorithms like token bucket and leaky bucket.

Rate Limiting Fundamentals

What you'll learn: How rate limiting protects services from overload and the core algorithms that govern traffic control.

What Is Rate Limiting?

Rate limiting controls how frequently a client or service can make requests within a time window. Think of it as a bouncer at a club: you can only enter so many times per hour, preventing any single person from overwhelming the venue.

Why Rate Limiting Matters

When services handle unbounded request volumes, they risk:

  • Resource exhaustion – memory, CPU, or connection pools depleting
  • Cascading failures – one overwhelmed service brings down others
  • Unfair resource allocation – aggressive clients starving others
  • Cost overruns – especially in pay-per-request cloud models

Rate limiting acts as a protective barrier, ensuring stability and fairness even under heavy load or attack.

Common Rate Limiting Algorithms

Token Bucket

Imagine a bucket that fills with tokens at a steady rate (say, 10 tokens/second). Each request consumes one token. If tokens are available, the request proceeds; otherwise, it's rejected or delayed. The bucket has a maximum capacity, allowing brief bursts when tokens accumulate during quiet periods.

Use case: Handles bursty traffic gracefully while maintaining average throughput.

Leaky Bucket

Water (requests) pours into a bucket with a small hole at the bottom. Water drains at a constant rate regardless of input speed. If the bucket overflows, excess requests are dropped. This smooths traffic into a steady stream.

Use case: Enforces strict, predictable output rates—ideal when downstream systems require constant flow.

Both algorithms integrate naturally with queue systems you've studied, where rate limiters control how quickly messages are dequeued or how many concurrent consumers process work.

Key Takeaway:
Rate limiting protects services from overload using algorithms like token bucket (allows bursts) and leaky bucket (enforces steady flow), ensuring system stability and fair resource distribution.