Key-Value Cache Fundamentals
You'll learn: How attention caching eliminates redundant computation during autoregressive LLM generation.
The Problem: Redundant Attention Computation
During autoregressive generation, your LLM predicts one token at a time. At each step, it must attend to all previous tokens in the sequence. Without caching, this means recalculating the attention keys and values for every past token at every generation step—an enormous waste.
Imagine you're writing a sentence word-by-word, and every time you add a new word, you re-read and re-annotate the entire sentence from scratch. That's what happens without a KV cache.
How KV Cache Works
In causal self-attention, each token's representation produces three vectors: Query (Q), Key (K), and Value (V). The crucial insight: keys and values for past tokens never change.
When generating token t, instead of recomputing K and V for tokens 1 through t-1, you:
- Store the K and V matrices from previous steps in memory
- Compute only the new token's K, V, and Q
- Append the new K and V to your cache
- Attend using the current Q against all cached keys and values
This transforms generation from O(n²) recomputation per token to O(n) with cached lookups.
The Trade-off
You're trading memory for speed. Each cached token stores two vectors (K and V) per attention head per layer. For a 7B model with 32 layers and 32 heads, this adds up fast—but it's typically worth it for the 10-100× speedup in generation.
Key Takeaway: The KV cache stores previously computed attention keys and values, eliminating redundant computation during sequential token generation and dramatically accelerating LLM inference at the cost of additional memory usage.