Course contentsShow
← Machine Learning and Deep Learning
Lesson 1185 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

What is Autoregressive Language Modeling?

Understanding the autoregressive approach: predicting the next token given all previous tokens in a sequence.

What is Autoregressive Language Modeling?

You'll learn: How autoregressive language models predict text one token at a time, using only previous context.

The Core Idea

Autoregressive language modeling is an approach where a model predicts the next token in a sequence based on all the tokens that came before it. The word "autoregressive" means the model uses its own previous outputs as inputs for future predictions.

Think of it like writing a sentence word by word: you choose each next word based on everything you've written so far. You can't peek ahead at future words—you only have the history to guide you.

How It Works

The model learns to estimate the probability of the next token given the sequence so far:

P(token_n | token_1, token_2, ..., token_{n-1})

During training, the model sees complete sequences but learns to predict each position using only the tokens before it. This is enforced through masked self-attention (which you learned about in lesson 1097), preventing the model from "cheating" by looking at future tokens.

Autoregressive Inference

At inference time, the model:

  1. Starts with a prompt or start token
  2. Predicts the most likely next token
  3. Appends that token to the sequence
  4. Repeats, using the growing sequence to predict further tokens

This is exactly the autoregressive inference pattern you studied in lesson 1100.

Why It Matters

This approach powers models like GPT, where the goal is generating text rather than just understanding it. The causal, left-to-right structure makes it natural for generation tasks.

Key Takeaway: Autoregressive language modeling predicts each token using only previous tokens in the sequence, enabling natural text generation through sequential, left-to-right prediction.