Concurrency Limiter Pattern
What You'll Learn
You'll learn how to limit the number of active concurrent requests being processed simultaneously, rather than controlling request rate over time.
The Core Idea
While rate limiting focuses on how many requests happen over time (e.g., 100 requests per minute), concurrency limiting focuses on how many requests are being processed right now.
Think of it like a parking lot: rate limiting controls how many cars can enter per hour, but concurrency limiting controls how many parking spots exist. Once all spots are full, new cars must wait—regardless of when they arrived.
Why Concurrency Limiting?
Some operations are resource-intensive in ways that don't align with time-based rate limits:
- Database connections: You have 100 connection pool slots
- Memory-heavy operations: Processing large file uploads
- CPU-bound tasks: Video transcoding, image processing
- External API calls: Third-party service limits concurrent connections
A service might handle 1000 requests/second for lightweight reads, but only 10 concurrent video transcoding jobs without crashing.
How It Works
- Track the count of in-flight requests (active processing)
- When a request arrives, check if current count < limit
- If under limit: increment counter, process request
- When request completes: decrement counter
- If at limit: reject (429 status) or queue the request
Unlike time-window algorithms (token bucket, sliding window), this is instantaneous state—you're counting what's happening right now, not aggregating over time.
Key Takeaway: Concurrency limiting protects resources by capping simultaneous active operations, complementing time-based rate limits for resource-intensive workloads.