Managing Concurrency Limits and Rate Limits
You'll learn how to use semaphores and rate limiting to respect API quotas while running as many parallel requests as safely possible.
The Balancing Act
In the previous lesson, you learned to send many requests in parallel—but providers don't allow unlimited simultaneous requests. Most APIs enforce:
- Concurrency limits: Maximum parallel requests at any moment (e.g., "10 requests at once")
- Rate limits: Maximum requests per time window (e.g., "500 requests per minute")
Ignore these, and you'll hit errors, waste retries, or get temporarily blocked. Respect them poorly, and you'll leave performance on the table.
Controlling Concurrency with Semaphores
A semaphore acts like a bouncer at a club: only N requests can execute simultaneously. When one finishes, the next waiting request gets a slot.
import asyncio
sem = asyncio.Semaphore(10) # Max 10 concurrent requests
async def call_llm_with_limit(prompt):
async with sem: # Acquire slot, block if all 10 busy
response = await llm_client.generate(prompt)
return response
This prevents overwhelming the API while still running 10 requests in parallel.
Rate Limiting Over Time
For per-minute quotas, use a token bucket or simple delay-based limiter. Libraries like aiolimiter help:
from aiolimiter import AsyncLimiter
limiter = AsyncLimiter(500, 60) # 500 requests per 60 seconds
async def call_with_rate_limit(prompt):
async with limiter:
return await llm_client.generate(prompt)
Combine both techniques to respect both concurrency and rate limits simultaneously.
Practical Strategy
Set your semaphore slightly below the provider's concurrency cap (e.g., 8 if limit is 10) to leave headroom. Monitor actual throughput and adjust. Test with bursts to find the sweet spot between speed and stability.
Key Takeaway:
Semaphores control how many requests run at once; rate limiters control requests over time—use both to maximize parallelism without hitting API quotas.