Teacher Forcing in Training
What you'll learn: How to speed up sequence model training by feeding the correct outputs as inputs during decoder training, rather than the model's own predictions.
The Problem Without Teacher Forcing
When training an encoder-decoder model (like for translation or captioning), the decoder generates one token at a time. Normally, each generated token becomes the input for generating the next token. But during early training, the decoder makes terrible predictions. If we feed these bad predictions back as input, the model spirals into increasingly worse outputs—it's learning from its own mistakes in a vicious cycle.
What Is Teacher Forcing?
Teacher forcing means using the ground truth token from the training data as the decoder's input at each time step, instead of the decoder's own prediction. It's like giving a student the correct answer to step 2 when teaching them step 3, rather than making them struggle through with their wrong answer from step 2.
The Training Process
During training with teacher forcing:
- At time step t, the decoder receives the actual correct token from position t-1
- The decoder predicts token t
- We compute loss against the ground truth
- Repeat for the next position
This allows the decoder to learn each position under optimal conditions, dramatically stabilizing and accelerating training.
The Exposure Bias Trade-off
The downside: at inference time, the model must use its own predictions (no ground truth available). This mismatch between training and testing is called exposure bias. Some advanced techniques mix teacher forcing with the model's predictions during training to reduce this gap.
Key Takeaway: Teacher forcing feeds ground truth tokens as decoder inputs during training to stabilize learning, though it creates a mismatch between training conditions and real-world inference.