Course contentsShow
← AI Engineering
Lesson 479 of 1,88612. Data Pipelines for AI SystemsFree lesson

Embedding API Rate Limits and Throttling

Handling API rate limits, implementing exponential backoff, and designing retry logic for embedding services.

Embedding API Rate Limits and Throttling

You'll learn how to handle API rate limits gracefully when generating embeddings at scale, so your pipeline doesn't crash or waste resources.

Why Rate Limits Matter

When you send documents to an embedding API (like OpenAI or Cohere), you're not working with unlimited bandwidth. Most services impose rate limits—maximum requests per minute or tokens per day. Hit that ceiling, and you'll get errors like 429 Too Many Requests.

Think of it like a highway toll booth: if too many cars arrive at once, some must wait. Your job is to manage that wait intelligently.

Core Strategies

Exponential Backoff

When you hit a rate limit, don't immediately retry—you'll just hit it again. Instead, wait progressively longer: 1 second, then 2, then 4, then 8. This gives the API time to reset while your system stays resilient.

import time

def embed_with_retry(texts, max_retries=5):
    for attempt in range(max_retries):
        try:
            return embedding_api.create(texts)
        except RateLimitError:
            wait_time = 2 ** attempt  # 1, 2, 4, 8, 16 seconds
            time.sleep(wait_time)
    raise Exception("Max retries exceeded")

Throttling

Proactively limit your request rate before hitting limits. If the API allows 60 requests/minute, send 50 to leave buffer room. Use delays between batches or queue-based processing.

Retry Logic

Design your pipeline to catch rate-limit errors specifically, distinguish them from other failures, and retry only when appropriate. Track failed items separately so you don't lose data.

Key Takeaway: Rate limits are inevitable at scale—handle them with exponential backoff, proactive throttling, and robust retry logic to build reliable embedding pipelines that gracefully handle API constraints.