Course contentsShow
← Machine Learning and Deep Learning
Lesson 1190 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Autoregressive Sampling at Inference

Sequential token generation during inference: feeding each predicted token back as input for the next step.

Autoregressive Sampling at Inference

What You'll Learn

You'll understand how autoregressive language models generate text one token at a time by feeding each prediction back as input for the next step.

The Core Idea

During inference (when generating text), an autoregressive model works like a storyteller who writes one word, then uses everything written so far to decide the next word, repeating this process until the story is complete.

Here's how it works step-by-step:

  1. Start with a prompt: You provide initial tokens like "The cat sat on"
  2. Predict the next token: The model processes this input and predicts "the" (with highest probability)
  3. Feed it back: Now your input becomes "The cat sat on the"
  4. Repeat: Predict the next token ("mat"), append it, making "The cat sat on the mat"
  5. Continue: Keep going until you hit a stopping condition (like an end token or maximum length)

This is called autoregressive sampling because each step depends on (regresses on) the model's own previous outputs.

The Key Mechanism

Unlike training (where you have the full target sequence), at inference you don't know what comes next—you must generate it. Each newly predicted token literally becomes part of the input sequence for the next prediction. The causal attention masking you learned earlier ensures the model only looks leftward, making this sequential generation possible.

Why "Sampling"?

The model outputs a probability distribution over all possible next tokens. You "sample" from this distribution (or pick the most likely token) to decide what comes next. Different sampling strategies produce different text.

Key Takeaway: Autoregressive sampling generates text sequentially by predicting one token, appending it to the input, and repeating—the model conditions each new prediction on all previously generated tokens.