Course contentsShow
← AI Engineering
Lesson 100 of 1,8863. API-First AI EngineeringFree lesson

Rate Limiting Basics

Understanding rate limits: requests per minute, tokens per minute, and how providers enforce usage quotas.

Rate Limiting Basics

What you'll learn: How API providers control usage through rate limits and why this matters for your AI applications.

What is Rate Limiting?

Rate limiting is how API providers enforce usage quotas to protect their infrastructure and ensure fair access. Think of it like a coffee shop that says "maximum 2 drinks per customer per hour" — the service is still available, but controlled to prevent abuse and maintain quality for everyone.

When you call an LLM API (OpenAI, Anthropic, Google, etc.), the provider tracks two primary limits:

Requests Per Minute (RPM)

This counts how many separate API calls you make in a 60-second window. Even if each request is small, hitting 100 requests in 10 seconds when your limit is 60 RPM will trigger a rejection.

Analogy: Like a checkout line — the store limits how many times you can get in line per hour, regardless of how much you buy each time.

Tokens Per Minute (TPM)

This measures the total volume of text (input + output tokens) processed across all your requests. You might stay under your RPM limit but exceed TPM if you're generating very long responses.

Analogy: Like a data plan — you have a bandwidth cap, not just a limit on number of connections.

How Enforcement Works

When you exceed a limit, the API returns an error (typically HTTP 429 "Too Many Requests"). Your application must handle this gracefully — often by waiting and retrying. Different providers have different tier limits based on your account status and payment plan.

Key Takeaway: Rate limits protect infrastructure through two metrics: requests per minute (call frequency) and tokens per minute (data volume). Understanding both is essential for building reliable AI applications that handle rejection errors appropriately.