Course contentsShow
← System Design
Lesson 1785 of 1,91943. Real-World System: Rate Limiter ServiceFree lesson

Non-Functional Requirements: Accuracy vs Performance

Balancing strict enforcement with throughput needs; accepting approximate counting for scale.

Non-Functional Requirements: Accuracy vs Performance

You'll learn: How rate limiters trade strict accuracy for throughput by using approximate counting at scale.

The Core Dilemma

In a rate limiter service, you face a fundamental tension: you can have perfect accuracy (never allow a single request over the limit) or high performance (handle millions of requests per second), but rarely both. Perfect counting requires synchronized, consistent state across all servers—which introduces coordination overhead that kills throughput.

At internet scale, strict enforcement becomes prohibitively expensive. Every rate-limit check would need to read and update a centralized counter, creating a bottleneck that limits your entire system's throughput.

Accepting Approximation

Most production rate limiters accept approximate counting. Instead of guaranteeing "exactly 100 requests per minute," they might allow 102-105 requests occasionally. This trades a small margin of error for massive performance gains.

Think of it like a highway toll booth: perfect accuracy means every car stops and waits for exact change. Approximate counting is like EZ-Pass—cars zip through with occasional read errors, but traffic flows smoothly.

Common Approximation Strategies

  • Local counters with eventual sync: Each server tracks requests locally, syncing periodically rather than real-time
  • Sliding window logs with sampling: Track recent requests approximately rather than storing every single one
  • Distributed counters without coordination: Accept that different servers might have slightly stale views

The key is choosing your accuracy threshold based on consequences. A banking API might need 99.9% accuracy; a social media feed rate limiter might accept 95% and prioritize speed.

Key Takeaway: Rate limiters at scale sacrifice perfect accuracy for performance, using approximate counting techniques to handle high throughput while keeping violations within acceptable bounds.