The KV Cache: Purpose and Benefits
What you'll learn: How the KV (key-value) cache dramatically speeds up autoregressive language model inference by eliminating redundant attention computations.
The Problem with Naive Autoregressive Generation
When a language model generates text token-by-token, each new token requires computing attention over all previous tokens. Without optimization, this means:
- Token 1: compute attention once
- Token 2: recompute attention for token 1, then compute for token 2
- Token 3: recompute attention for tokens 1 and 2, then compute for token 3
- ...and so on
This creates massive redundancy—you're recalculating the same attention keys and values for earlier tokens over and over.
What is KV Caching?
The KV cache stores the computed key and value matrices from the attention mechanism for all previously generated tokens. When generating the next token, instead of recomputing attention from scratch, you:
- Retrieve cached keys and values for all previous tokens
- Compute keys and values only for the new token
- Perform attention using the cached + new KV pairs
- Cache the new token's keys and values for the next iteration
The Trade-Off
Benefits:
- Speed: Reduces time complexity from O(n²) to O(n) per token generation
- Efficiency: Eliminates redundant matrix multiplications across transformer layers
- Throughput: Enables faster batch processing and lower latency
Cost:
- Memory: Must store keys and values for every token, across all layers and attention heads
- Scaling: Memory usage grows linearly with sequence length and batch size
Think of it like showing your ID at a building entrance once and getting a pass, versus showing ID every single time you walk through.
Key Takeaway: The KV cache trades memory for speed, storing intermediate attention computations to avoid recalculating them during autoregressive generation—essential for practical LLM serving.