Course contentsShow
← Machine Learning and Deep Learning
Lesson 1671 of 3,53836. LLM Inference OptimizationFree lesson

Prefill vs Decode Phase Dynamics

Understanding compute-bound prefill and memory-bound decode phases in autoregressive generation.

Prefill vs Decode Phase Dynamics

You'll learn: Why LLM inference splits into two distinct phases with fundamentally different computational characteristics.

The Two Faces of Autoregressive Generation

When an LLM generates text, it operates in two distinct phases:

Prefill Phase (Compute-Bound)

During prefill, the model processes your entire input prompt in one parallel forward pass. All tokens attend to each other simultaneously, filling the KV cache with key-value pairs for every input token. This phase is compute-bound — the bottleneck is performing massive matrix multiplications across all attention heads and layers. Your GPU cores are working hard, crunching numbers at full utilization.

Think of it like reading an entire book chapter at once to understand context — you process all words simultaneously.

Decode Phase (Memory-Bound)

Once prefill completes, decode generates tokens one at a time. Each new token attends to all previous tokens by reading from the KV cache, then adds its own KV pair. Since you're only computing attention for a single token, you're doing far fewer operations. The bottleneck shifts to memory bandwidth — constantly fetching the growing KV cache from memory (HBM) to compute attention scores.

This is like repeatedly skimming back through that entire chapter to write each new sentence, one word at a time — most time is spent "looking back" rather than "thinking."

Why This Matters

These opposing bottlenecks require different optimization strategies. Prefill benefits from larger batch sizes and computational optimizations (Flash Attention). Decode benefits from memory compression (quantization, KV cache pruning) and reducing data movement. Understanding this dichotomy is essential for efficient inference system design.

Key Takeaway: Prefill is compute-bound (parallel processing of input), while decode is memory-bound (sequential generation reading from cache) — optimize each phase differently.