What is Autoregressive Language Modeling?
You'll learn: How autoregressive language models predict text one token at a time, using only previous context.
The Core Idea
Autoregressive language modeling is an approach where a model predicts the next token in a sequence based on all the tokens that came before it. The word "autoregressive" means the model uses its own previous outputs as inputs for future predictions.
Think of it like writing a sentence word by word: you choose each next word based on everything you've written so far. You can't peek ahead at future words—you only have the history to guide you.
How It Works
The model learns to estimate the probability of the next token given the sequence so far:
P(token_n | token_1, token_2, ..., token_{n-1})
During training, the model sees complete sequences but learns to predict each position using only the tokens before it. This is enforced through masked self-attention (which you learned about in lesson 1097), preventing the model from "cheating" by looking at future tokens.
Autoregressive Inference
At inference time, the model:
- Starts with a prompt or start token
- Predicts the most likely next token
- Appends that token to the sequence
- Repeats, using the growing sequence to predict further tokens
This is exactly the autoregressive inference pattern you studied in lesson 1100.
Why It Matters
This approach powers models like GPT, where the goal is generating text rather than just understanding it. The causal, left-to-right structure makes it natural for generation tasks.
Key Takeaway: Autoregressive language modeling predicts each token using only previous tokens in the sequence, enabling natural text generation through sequential, left-to-right prediction.