Autoregressive Sampling at Inference
What You'll Learn
You'll understand how autoregressive language models generate text one token at a time by feeding each prediction back as input for the next step.
The Core Idea
During inference (when generating text), an autoregressive model works like a storyteller who writes one word, then uses everything written so far to decide the next word, repeating this process until the story is complete.
Here's how it works step-by-step:
- Start with a prompt: You provide initial tokens like "The cat sat on"
- Predict the next token: The model processes this input and predicts "the" (with highest probability)
- Feed it back: Now your input becomes "The cat sat on the"
- Repeat: Predict the next token ("mat"), append it, making "The cat sat on the mat"
- Continue: Keep going until you hit a stopping condition (like an end token or maximum length)
This is called autoregressive sampling because each step depends on (regresses on) the model's own previous outputs.
The Key Mechanism
Unlike training (where you have the full target sequence), at inference you don't know what comes next—you must generate it. Each newly predicted token literally becomes part of the input sequence for the next prediction. The causal attention masking you learned earlier ensures the model only looks leftward, making this sequential generation possible.
Why "Sampling"?
The model outputs a probability distribution over all possible next tokens. You "sample" from this distribution (or pick the most likely token) to decide what comes next. Different sampling strategies produce different text.
Key Takeaway: Autoregressive sampling generates text sequentially by predicting one token, appending it to the input, and repeating—the model conditions each new prediction on all previously generated tokens.