Course contentsShow
← AI Engineering
Lesson 988 of 1,88624. Building AI APIsFree lesson

Rate Limiting Fundamentals

Understanding rate limiting strategies: fixed window, sliding window, and token bucket algorithms.

Rate Limiting Fundamentals

What you'll learn: How to protect your AI API from being overwhelmed using three core rate limiting algorithms.

Why Rate Limiting Matters

Rate limiting controls how many requests a user can make to your API within a time window. Without it, a single user could consume all your LLM tokens, rack up massive costs, or crash your service. It's essential protection for any production AI API.

Three Core Strategies

Fixed Window

Think of this as hourly parking zones. Each hour is a bucket—you get 100 requests per hour. At 2:00 PM, you can make 100 requests. When the clock hits 3:00 PM, your counter resets to zero, and you get another 100.

Problem: A user could game the system by making 100 requests at 2:59 PM and another 100 at 3:00 PM—200 requests in two minutes.

Sliding Window

This is like a rolling 60-minute window that moves with each request. Instead of resetting at fixed intervals, it looks back at the last hour from right now. If you made 100 requests in the past 60 minutes, you're blocked until earlier requests "age out."

Benefit: Smooths out traffic bursts and prevents boundary exploits.

Token Bucket

Imagine a bucket that holds tokens (say, 100). Each request costs one token. The bucket refills at a steady rate (maybe 10 tokens per minute). You can burst up to 100 requests quickly if you've saved tokens, but sustained high usage will drain your bucket.

Benefit: Allows controlled bursts while enforcing long-term limits—ideal for LLM APIs where quick follow-ups are common but sustained abuse must be prevented.

Choosing Your Strategy

  • Fixed window: Simplest to implement, works for basic protection
  • Sliding window: Best for fair, consistent rate limiting
  • Token bucket: Best for AI APIs with bursty, conversational patterns

Key Takeaway: Rate limiting protects your AI API from abuse and cost overruns—use token buckets for conversational AI, sliding windows for fairness, and fixed windows for simplicity.