Teacher Forcing in Autoregressive Training
What You'll Learn
You'll learn how teacher forcing accelerates autoregressive model training by feeding ground-truth tokens instead of model predictions during each step.
The Core Idea
When training an autoregressive language model (like GPT), you need the model to predict the next token given all previous tokens. But here's the challenge: if you waited for the model to generate token 2 before predicting token 3, training would be painfully slow—you'd have to process sequences one token at a time.
Teacher forcing solves this by "cheating" during training: instead of feeding the model's own predictions as input for the next step, we feed it the actual ground-truth tokens from our training data.
How It Works
Imagine teaching someone to complete the sentence "The cat sat on the ___".
- Without teacher forcing: Student guesses "mat", then you ask them to continue from "The cat sat on the mat..." (using their own guess)
- With teacher forcing: Student guesses "mat", but you show them the correct answer was "rug", and ask them to continue from "The cat sat on the rug..." (using the truth)
In practice, this means during training on the sentence "Hello world", the model:
- Sees
[H]→ predictse - Sees
[H, e]→ predictsl - Sees
[H, e, l]→ predictsl
All these predictions happen in parallel because we know all the ground-truth inputs ahead of time. The causal attention mask (from lesson 1187) ensures each position only attends to previous positions, maintaining the autoregressive property.
Why It Matters
Teacher forcing enables parallel loss computation across all tokens in a sequence simultaneously, making training hundreds of times faster than sequential generation would be.
Key Takeaway: Teacher forcing uses ground-truth tokens as input during training, allowing parallel computation of all next-token predictions at once rather than waiting for sequential generation.