KV Cache Eviction Strategies
What you'll learn: How to intelligently decide which cached key-value pairs to drop when memory runs out during LLM inference.
The Memory Management Challenge
You've learned that KV cache stores attention keys and values to avoid recomputing them during autoregressive generation. But what happens when your cache fills up? With long sequences or limited memory, you can't keep everything forever.
KV cache eviction is the process of selectively removing cached positions when you hit memory limits, keeping only the most valuable information.
Core Eviction Strategies
Random Eviction
Drop random positions from the cache. Simple but wasteful—you might discard critical context.
Recent-Only (FIFO)
Keep only the most recent N tokens. Works well for local context but loses important information from earlier in the sequence (like the original question in a long conversation).
H₂O (Heavy Hitters On)
Track which cached positions receive the most attention during generation. Evict positions with the lowest cumulative attention scores—they're contributing less to predictions.
Sliding Window + Anchors
Combine sliding window attention (keep recent tokens) with "anchor" tokens that are never evicted—typically special tokens, the initial prompt, or positions with historically high attention.
The Trade-off
Think of it like a library with limited shelf space: you must choose which books to keep accessible. Random removal is chaotic; keeping only new arrivals ignores classics; tracking which books get checked out most (attention scores) is smarter.
Key Takeaway: KV cache eviction strategies balance memory constraints with generation quality by intelligently dropping less-important cached positions based on recency, attention patterns, or strategic anchoring rather than keeping everything or discarding randomly.