Next-Token Prediction Loss
What you'll learn: How we measure and optimize the error in predicting the next token during autoregressive language model training.
The Core Idea
When training an autoregressive language model (like GPT), the model predicts the next token at each position in a sequence. But how do we quantify how wrong those predictions are? That's where next-token prediction loss comes in.
At each position, your model outputs a probability distribution over the entire vocabulary (think: 50,000+ possible words or subwords). The loss measures how far that predicted distribution is from the correct answer—which token actually comes next.
Cross-Entropy Loss Over Vocabulary
We use cross-entropy loss for this task. Here's the intuition:
- Your model assigns a probability to every possible next token
- The true next token is known (from your training data)
- Cross-entropy penalizes predictions that assign low probability to the correct token
Analogy: Imagine a multiple-choice test with 50,000 options. You write down confidence scores for each answer. Cross-entropy measures how confident you were in the correct answer. If you gave it 90% confidence, great! If only 1%, that's a high loss.
The Formula in Plain English
For each position in the sequence:
- Model outputs probability for each vocabulary word
- Compare to the one-hot encoded correct next token
- Compute
-log(probability of correct token) - Average across all positions in all sequences
The model learns by minimizing this loss, gradually assigning higher probability to correct next tokens.
Why It Works
Cross-entropy naturally pushes the model to be confident and correct. A probability of 0.5 on the right answer is better than 0.1, but 0.99 is better still.
Key Takeaway: Next-token prediction loss uses cross-entropy to measure how well the model's probability distribution matches the true next token at every position, driving the model to predict accurately across the entire vocabulary.