Course contentsShow
← Machine Learning and Deep Learning
Lesson 1678 of 3,53836. LLM Inference OptimizationFree lesson

KV Cache Eviction Strategies

Intelligently dropping less important cached positions when memory limits are reached.

KV Cache Eviction Strategies

What you'll learn: How to intelligently decide which cached key-value pairs to drop when memory runs out during LLM inference.

The Memory Management Challenge

You've learned that KV cache stores attention keys and values to avoid recomputing them during autoregressive generation. But what happens when your cache fills up? With long sequences or limited memory, you can't keep everything forever.

KV cache eviction is the process of selectively removing cached positions when you hit memory limits, keeping only the most valuable information.

Core Eviction Strategies

Random Eviction

Drop random positions from the cache. Simple but wasteful—you might discard critical context.

Recent-Only (FIFO)

Keep only the most recent N tokens. Works well for local context but loses important information from earlier in the sequence (like the original question in a long conversation).

H₂O (Heavy Hitters On)

Track which cached positions receive the most attention during generation. Evict positions with the lowest cumulative attention scores—they're contributing less to predictions.

Sliding Window + Anchors

Combine sliding window attention (keep recent tokens) with "anchor" tokens that are never evicted—typically special tokens, the initial prompt, or positions with historically high attention.

The Trade-off

Think of it like a library with limited shelf space: you must choose which books to keep accessible. Random removal is chaotic; keeping only new arrivals ignores classics; tracking which books get checked out most (attention scores) is smarter.

Key Takeaway: KV cache eviction strategies balance memory constraints with generation quality by intelligently dropping less-important cached positions based on recency, attention patterns, or strategic anchoring rather than keeping everything or discarding randomly.