Local Caching for Performance
You'll learn how to combine local in-memory counters with distributed state to dramatically reduce Redis calls and improve latency in your rate limiter.
The Latency Problem
Every rate-limit check that hits Redis adds network latency—typically 1-5ms per call. At high throughput (millions of requests/second), these round-trips become a bottleneck. The solution? Keep a local cache of counters on each application server.
How Local Caching Works
Think of it like keeping a running tab at a coffee shop instead of swiping your card for every sip. Each server maintains in-memory counters (e.g., a hash map of user_id → {count, window_start}). When a request arrives:
- Check locally first: If the counter exists and is within the current time window, increment it in-memory (nanoseconds, not milliseconds)
- Periodically sync to Redis: Flush accumulated counts every 100-500ms, or when local count reaches a threshold
- Pull distributed state: On cache miss or window expiration, fetch the authoritative count from Redis
The Trade-Off
Local caching trades perfect accuracy for speed. If one server allows 90 requests and another allows 95 before syncing, a user might briefly exceed their 100-request limit. This is acceptable for most rate limiters where "soft limits" (within 5-10% variance) don't compromise system safety.
When to Use It
Best for high-traffic, low-consequence endpoints (e.g., API rate limits) rather than critical security checks (e.g., payment fraud prevention). Combine with background sync and TTL-based eviction to keep memory bounded.
Key Takeaway:
Local caching reduces Redis calls by 90%+, slashing latency from milliseconds to microseconds—at the cost of slight temporary inaccuracy across distributed nodes.