The Autoregressive Generation Bottleneck
What you'll learn: Why generating text token-by-token in LLMs creates fundamental computational inefficiencies and memory pressure during inference.
The Sequential Generation Problem
When LLMs generate text, they produce one token at a time in an autoregressive manner—each new token depends on all previous tokens. This creates a bottleneck: unlike training (where the entire sequence is known), inference must happen sequentially. You can't generate token 50 until you've generated tokens 1-49.
Why This Creates Redundancy
Think of it like repeatedly reading an entire book from the beginning just to write the next sentence. For each new token:
- The model must load all previous tokens' key-value pairs from memory
- Attention is computed over the entire growing context
- The same earlier computations get repeated with minimal new information
As your sequence grows from 10 to 100 to 1,000 tokens, each step requires more memory bandwidth and compute—but you're only generating one new token per forward pass.
Memory Challenges
The KV cache (storing keys and values from attention layers) grows linearly with sequence length and batch size. For a 7B parameter model with a 2,048 token context:
- Each token adds ~0.5MB to the KV cache
- Full context = ~1GB just for cached attention states
- This memory is accessed repeatedly, creating bandwidth bottlenecks
Unlike the model weights (loaded once), the KV cache must be read and updated every generation step.
Key Takeaway: Autoregressive generation forces LLMs to process tokens sequentially rather than in parallel, creating computational redundancy where the same context is re-processed repeatedly, while the growing KV cache consumes memory and bandwidth proportional to sequence length.