Embedding API Rate Limits and Throttling
You'll learn how to handle API rate limits gracefully when generating embeddings at scale, so your pipeline doesn't crash or waste resources.
Why Rate Limits Matter
When you send documents to an embedding API (like OpenAI or Cohere), you're not working with unlimited bandwidth. Most services impose rate limits—maximum requests per minute or tokens per day. Hit that ceiling, and you'll get errors like 429 Too Many Requests.
Think of it like a highway toll booth: if too many cars arrive at once, some must wait. Your job is to manage that wait intelligently.
Core Strategies
Exponential Backoff
When you hit a rate limit, don't immediately retry—you'll just hit it again. Instead, wait progressively longer: 1 second, then 2, then 4, then 8. This gives the API time to reset while your system stays resilient.
import time
def embed_with_retry(texts, max_retries=5):
for attempt in range(max_retries):
try:
return embedding_api.create(texts)
except RateLimitError:
wait_time = 2 ** attempt # 1, 2, 4, 8, 16 seconds
time.sleep(wait_time)
raise Exception("Max retries exceeded")
Throttling
Proactively limit your request rate before hitting limits. If the API allows 60 requests/minute, send 50 to leave buffer room. Use delays between batches or queue-based processing.
Retry Logic
Design your pipeline to catch rate-limit errors specifically, distinguish them from other failures, and retry only when appropriate. Track failed items separately so you don't lose data.
Key Takeaway: Rate limits are inevitable at scale—handle them with exponential backoff, proactive throttling, and robust retry logic to build reliable embedding pipelines that gracefully handle API constraints.