Prefill vs Decode Phase Dynamics
You'll learn: Why LLM inference splits into two distinct phases with fundamentally different computational characteristics.
The Two Faces of Autoregressive Generation
When an LLM generates text, it operates in two distinct phases:
Prefill Phase (Compute-Bound)
During prefill, the model processes your entire input prompt in one parallel forward pass. All tokens attend to each other simultaneously, filling the KV cache with key-value pairs for every input token. This phase is compute-bound — the bottleneck is performing massive matrix multiplications across all attention heads and layers. Your GPU cores are working hard, crunching numbers at full utilization.
Think of it like reading an entire book chapter at once to understand context — you process all words simultaneously.
Decode Phase (Memory-Bound)
Once prefill completes, decode generates tokens one at a time. Each new token attends to all previous tokens by reading from the KV cache, then adds its own KV pair. Since you're only computing attention for a single token, you're doing far fewer operations. The bottleneck shifts to memory bandwidth — constantly fetching the growing KV cache from memory (HBM) to compute attention scores.
This is like repeatedly skimming back through that entire chapter to write each new sentence, one word at a time — most time is spent "looking back" rather than "thinking."
Why This Matters
These opposing bottlenecks require different optimization strategies. Prefill benefits from larger batch sizes and computational optimizations (Flash Attention). Decode benefits from memory compression (quantization, KV cache pruning) and reducing data movement. Understanding this dichotomy is essential for efficient inference system design.
Key Takeaway: Prefill is compute-bound (parallel processing of input), while decode is memory-bound (sequential generation reading from cache) — optimize each phase differently.