The Autoregressive Bottleneck in LLM Inference
What you'll learn: Why LLM inference is fundamentally memory-bound, not compute-bound, and how sequential token generation creates the primary throughput barrier.
The Sequential Nature Problem
Large Language Models generate text one token at a time in an autoregressive manner—each new token depends on all previous tokens. Unlike training where you can process entire sequences in parallel, inference must wait for token N before generating token N+1. This creates an unavoidable serialization bottleneck.
Memory Bandwidth, Not Compute
Here's the counterintuitive insight: modern GPUs sit idle during LLM inference.
The bottleneck isn't matrix multiplication speed—it's how fast you can load model weights from memory. For each token generation step, you must:
- Load billions of parameters from GPU memory (HBM)
- Perform relatively small computations (one forward pass)
- Write a tiny output (next token logits)
- Repeat for the next token
Analogy: Imagine a chef (GPU compute) who can chop vegetables instantly, but must walk to a distant pantry (memory) to fetch each ingredient one at a time. The chef spends most time walking, not cooking.
Why This Matters
- Batch size helps less than in training—you're still limited by how fast you can stream weights
- Faster GPUs (more FLOPS) don't proportionally improve generation speed
- Memory bandwidth (GB/s) becomes the critical specification
- Each generated token requires loading the entire model from memory again
Key Takeaway: LLM inference throughput is fundamentally limited by memory bandwidth due to sequential token generation—the GPU waits for data, not computation. This is why techniques like continuous batching and KV cache optimization focus on maximizing memory efficiency rather than compute utilization.