Autoregressive Inference
What you'll learn: How transformer decoders generate text one token at a time during inference, building each prediction on all previously generated tokens.
The Core Idea
During training, you used teacher forcing where the decoder saw the entire target sequence at once (with masking). But at inference time—when generating actual outputs—you don't have the target sequence yet! The model must create it from scratch.
Autoregressive inference means the decoder generates output sequentially: it produces one token, then uses that token as input to generate the next token, then uses both previous tokens to generate the third, and so on. Each prediction is conditioned on (depends on) all tokens generated so far.
How It Works Step-by-Step
- Start: Begin with a special start token (like
<BOS>) - Generate token 1: Decoder processes the start token and outputs a probability distribution over your vocabulary. Sample or select the most likely token.
- Generate token 2: Feed the start token and token 1 back into the decoder. Generate token 2.
- Repeat: Continue feeding all previous tokens back in, generating one new token each step
- Stop: End when you generate an end token (
<EOS>) or reach maximum length
The Analogy
Think of writing a story one word at a time where you can only see what you've written so far—no peeking ahead. Each new word depends on reading everything you've already written.
Key Takeaway: Autoregressive inference is the sequential, one-token-at-a-time generation process where each new prediction conditions on all previously generated outputs, making it inherently slower than the parallel training phase.