Scaling Rate Limiter Horizontally
What you'll learn: How to distribute rate limiter workload across multiple nodes and use consistent hashing to route tenant requests efficiently as demand grows.
Why Scale Horizontally?
As your user base and request volume grow, a single rate limiter instance becomes a bottleneck. Horizontal scaling means adding more rate limiter nodes to handle increased load, rather than upgrading a single machine (vertical scaling). The challenge is ensuring that requests from the same tenant consistently reach the same node—otherwise, you lose accurate count tracking.
Consistent Hashing for Tenant Routing
When multiple rate limiter instances exist, you need a routing strategy. Consistent hashing maps each tenant ID to a specific rate limiter node. Hash the tenant identifier (like user_id or api_key), and the result determines which node handles that tenant's requests.
Analogy: Think of it like mail sorting at a post office. Each zip code (tenant) is assigned to a specific sorting station (node). All mail for that zip code always goes to the same station, ensuring accurate counting and processing.
Benefits of This Approach
- Linear scalability: Add nodes as traffic grows
- Tenant affinity: All requests for a tenant hit the same node and Redis shard
- Load distribution: Tenants spread evenly across nodes
- Isolation: One tenant's burst doesn't overwhelm resources for others
Implementation Considerations
Your load balancer or API gateway performs the hash and routes accordingly. When adding/removing nodes, consistent hashing minimizes remapping—only a fraction of tenants move to different nodes, not all of them.
Key Takeaway: Horizontal scaling with consistent hashing allows your rate limiter to grow by distributing tenant workload across multiple nodes while maintaining accurate per-tenant rate tracking through deterministic routing.