Course contentsShow
← Machine Learning and Deep Learning
Lesson 1667 of 3,53836. LLM Inference OptimizationFree lesson

The Autoregressive Generation Bottleneck

Why token-by-token generation creates computational redundancy and memory challenges in LLM inference.

The Autoregressive Generation Bottleneck

What you'll learn: Why generating text token-by-token in LLMs creates fundamental computational inefficiencies and memory pressure during inference.

The Sequential Generation Problem

When LLMs generate text, they produce one token at a time in an autoregressive manner—each new token depends on all previous tokens. This creates a bottleneck: unlike training (where the entire sequence is known), inference must happen sequentially. You can't generate token 50 until you've generated tokens 1-49.

Why This Creates Redundancy

Think of it like repeatedly reading an entire book from the beginning just to write the next sentence. For each new token:

  1. The model must load all previous tokens' key-value pairs from memory
  2. Attention is computed over the entire growing context
  3. The same earlier computations get repeated with minimal new information

As your sequence grows from 10 to 100 to 1,000 tokens, each step requires more memory bandwidth and compute—but you're only generating one new token per forward pass.

Memory Challenges

The KV cache (storing keys and values from attention layers) grows linearly with sequence length and batch size. For a 7B parameter model with a 2,048 token context:

  • Each token adds ~0.5MB to the KV cache
  • Full context = ~1GB just for cached attention states
  • This memory is accessed repeatedly, creating bandwidth bottlenecks

Unlike the model weights (loaded once), the KV cache must be read and updated every generation step.

Key Takeaway: Autoregressive generation forces LLMs to process tokens sequentially rather than in parallel, creating computational redundancy where the same context is re-processed repeatedly, while the growing KV cache consumes memory and bandwidth proportional to sequence length.