Course contentsShow
← Machine Learning and Deep Learning
Lesson 2991 of 3,53865. LLM Inference EnginesFree lesson

The Autoregressive Bottleneck in LLM Inference

Understanding why sequential token generation limits LLM throughput and how generation speed depends on memory bandwidth, not compute.

The Autoregressive Bottleneck in LLM Inference

What you'll learn: Why LLM inference is fundamentally memory-bound, not compute-bound, and how sequential token generation creates the primary throughput barrier.

The Sequential Nature Problem

Large Language Models generate text one token at a time in an autoregressive manner—each new token depends on all previous tokens. Unlike training where you can process entire sequences in parallel, inference must wait for token N before generating token N+1. This creates an unavoidable serialization bottleneck.

Memory Bandwidth, Not Compute

Here's the counterintuitive insight: modern GPUs sit idle during LLM inference.

The bottleneck isn't matrix multiplication speed—it's how fast you can load model weights from memory. For each token generation step, you must:

  1. Load billions of parameters from GPU memory (HBM)
  2. Perform relatively small computations (one forward pass)
  3. Write a tiny output (next token logits)
  4. Repeat for the next token

Analogy: Imagine a chef (GPU compute) who can chop vegetables instantly, but must walk to a distant pantry (memory) to fetch each ingredient one at a time. The chef spends most time walking, not cooking.

Why This Matters

  • Batch size helps less than in training—you're still limited by how fast you can stream weights
  • Faster GPUs (more FLOPS) don't proportionally improve generation speed
  • Memory bandwidth (GB/s) becomes the critical specification
  • Each generated token requires loading the entire model from memory again

Key Takeaway: LLM inference throughput is fundamentally limited by memory bandwidth due to sequential token generation—the GPU waits for data, not computation. This is why techniques like continuous batching and KV cache optimization focus on maximizing memory efficiency rather than compute utilization.