Course contentsShow
← Machine Learning and Deep Learning
Lesson 2921 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Cache Eviction Policies

Implementing LRU, LFU, and TTL-based eviction strategies to manage cache memory efficiently in production systems.

Cache Eviction Policies

What you'll learn: How to implement LRU, LFU, and TTL-based eviction strategies to keep your inference cache lean and effective.

Why Cache Eviction Matters

Once you've implemented result caching for model predictions, your cache will eventually fill up. Without a smart eviction policy, you'll either run out of memory or keep stale predictions indefinitely. Cache eviction policies automatically decide which entries to remove when space runs low or data becomes outdated.

Think of it like a bookshelf: when it's full, you need rules for which books to remove—the ones you haven't read in years, the ones you rarely open, or simply the oldest ones.

Three Core Eviction Strategies

LRU (Least Recently Used)

Removes entries that haven't been accessed in the longest time. Best when recent predictions are likely to be requested again (temporal locality). When your cache is full and a new prediction arrives, the entry with the oldest last-access timestamp gets evicted.

LFU (Least Frequently Used)

Tracks how many times each cache entry has been requested and evicts the least popular ones. Ideal when certain inputs (like common user queries) appear repeatedly over time, while others are one-offs. Maintains access counters per cache key.

TTL (Time-To-Live)

Assigns each cached result an expiration timestamp. After the TTL expires (e.g., 5 minutes), the entry is automatically removed regardless of usage. Perfect for scenarios where model predictions become stale or when your model is retrained regularly and old predictions shouldn't persist.

Choosing Your Strategy

  • LRU: General-purpose, works well for most inference workloads with predictable access patterns
  • LFU: High-traffic APIs with skewed request distributions ("power law" behavior)
  • TTL: Model versioning scenarios, time-sensitive predictions, or compliance requirements

Key Takeaway: Cache eviction policies—LRU, LFU, and TTL—prevent memory overflow and stale results by intelligently removing entries based on recency, frequency, or age, ensuring your inference cache remains efficient in production.