Course contentsShow
← Machine Learning and Deep Learning
Lesson 1189 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Next-Token Prediction Loss

Formulating the cross-entropy loss over vocabulary for each position in the autoregressive objective.

Next-Token Prediction Loss

What you'll learn: How we measure and optimize the error in predicting the next token during autoregressive language model training.

The Core Idea

When training an autoregressive language model (like GPT), the model predicts the next token at each position in a sequence. But how do we quantify how wrong those predictions are? That's where next-token prediction loss comes in.

At each position, your model outputs a probability distribution over the entire vocabulary (think: 50,000+ possible words or subwords). The loss measures how far that predicted distribution is from the correct answer—which token actually comes next.

Cross-Entropy Loss Over Vocabulary

We use cross-entropy loss for this task. Here's the intuition:

  • Your model assigns a probability to every possible next token
  • The true next token is known (from your training data)
  • Cross-entropy penalizes predictions that assign low probability to the correct token

Analogy: Imagine a multiple-choice test with 50,000 options. You write down confidence scores for each answer. Cross-entropy measures how confident you were in the correct answer. If you gave it 90% confidence, great! If only 1%, that's a high loss.

The Formula in Plain English

For each position in the sequence:

  1. Model outputs probability for each vocabulary word
  2. Compare to the one-hot encoded correct next token
  3. Compute -log(probability of correct token)
  4. Average across all positions in all sequences

The model learns by minimizing this loss, gradually assigning higher probability to correct next tokens.

Why It Works

Cross-entropy naturally pushes the model to be confident and correct. A probability of 0.5 on the right answer is better than 0.1, but 0.99 is better still.

Key Takeaway: Next-token prediction loss uses cross-entropy to measure how well the model's probability distribution matches the true next token at every position, driving the model to predict accurately across the entire vocabulary.