Cache Eviction Policies
What you'll learn: How to implement LRU, LFU, and TTL-based eviction strategies to keep your inference cache lean and effective.
Why Cache Eviction Matters
Once you've implemented result caching for model predictions, your cache will eventually fill up. Without a smart eviction policy, you'll either run out of memory or keep stale predictions indefinitely. Cache eviction policies automatically decide which entries to remove when space runs low or data becomes outdated.
Think of it like a bookshelf: when it's full, you need rules for which books to remove—the ones you haven't read in years, the ones you rarely open, or simply the oldest ones.
Three Core Eviction Strategies
LRU (Least Recently Used)
Removes entries that haven't been accessed in the longest time. Best when recent predictions are likely to be requested again (temporal locality). When your cache is full and a new prediction arrives, the entry with the oldest last-access timestamp gets evicted.
LFU (Least Frequently Used)
Tracks how many times each cache entry has been requested and evicts the least popular ones. Ideal when certain inputs (like common user queries) appear repeatedly over time, while others are one-offs. Maintains access counters per cache key.
TTL (Time-To-Live)
Assigns each cached result an expiration timestamp. After the TTL expires (e.g., 5 minutes), the entry is automatically removed regardless of usage. Perfect for scenarios where model predictions become stale or when your model is retrained regularly and old predictions shouldn't persist.
Choosing Your Strategy
- LRU: General-purpose, works well for most inference workloads with predictable access patterns
- LFU: High-traffic APIs with skewed request distributions ("power law" behavior)
- TTL: Model versioning scenarios, time-sensitive predictions, or compliance requirements
Key Takeaway: Cache eviction policies—LRU, LFU, and TTL—prevent memory overflow and stale results by intelligently removing entries based on recency, frequency, or age, ensuring your inference cache remains efficient in production.