Non-Functional Requirements: Latency and Availability
What you'll learn: Why a rate limiter must be lightning-fast and never become the bottleneck that breaks your system.
The Critical Performance Challenge
A rate limiter sits in the critical request path—every single API call must pass through it before reaching your actual service. If the rate limiter adds significant latency or becomes unavailable, you've created a single point of failure that defeats the entire purpose of protecting your system.
Latency: The Sub-Millisecond Imperative
Target: < 1ms overhead
Think of a rate limiter like a security checkpoint at a concert entrance. If checking each ticket takes 30 seconds, the line backs up and the experience is ruined. Your rate limiter must make allow/deny decisions in under 1 millisecond to avoid degrading user experience.
This requirement drives architectural choices:
- In-memory storage (Redis, not database queries)
- Co-located deployment (same data center as API servers)
- Simple algorithms (token bucket, not complex ML models)
- Minimal network hops
Availability: Never Block Legitimate Traffic
Target: 99.99%+ uptime
When a rate limiter fails, you face a terrible choice: fail open (allow all traffic, losing protection) or fail closed (block all traffic, losing availability). Neither is acceptable.
High availability requires:
- Redundant instances across multiple availability zones
- Fail-open defaults when rate limiter is unreachable
- Circuit breakers to detect and route around failures
- Local caching of rate limit decisions for graceful degradation
Key Takeaway: A rate limiter must be faster than the services it protects and more available than the system it guards—any added latency or downtime transforms it from protection into liability.