Course contentsShow
← Machine Learning and Deep Learning
Lesson 1029 of 3,53823. Recurrent Neural NetworksFree lesson

Teacher Forcing in Training

Using ground truth tokens as decoder inputs during training to stabilize learning and accelerate convergence.

Teacher Forcing in Training

What you'll learn: How to speed up sequence model training by feeding the correct outputs as inputs during decoder training, rather than the model's own predictions.

The Problem Without Teacher Forcing

When training an encoder-decoder model (like for translation or captioning), the decoder generates one token at a time. Normally, each generated token becomes the input for generating the next token. But during early training, the decoder makes terrible predictions. If we feed these bad predictions back as input, the model spirals into increasingly worse outputs—it's learning from its own mistakes in a vicious cycle.

What Is Teacher Forcing?

Teacher forcing means using the ground truth token from the training data as the decoder's input at each time step, instead of the decoder's own prediction. It's like giving a student the correct answer to step 2 when teaching them step 3, rather than making them struggle through with their wrong answer from step 2.

The Training Process

During training with teacher forcing:

  1. At time step t, the decoder receives the actual correct token from position t-1
  2. The decoder predicts token t
  3. We compute loss against the ground truth
  4. Repeat for the next position

This allows the decoder to learn each position under optimal conditions, dramatically stabilizing and accelerating training.

The Exposure Bias Trade-off

The downside: at inference time, the model must use its own predictions (no ground truth available). This mismatch between training and testing is called exposure bias. Some advanced techniques mix teacher forcing with the model's predictions during training to reduce this gap.

Key Takeaway: Teacher forcing feeds ground truth tokens as decoder inputs during training to stabilize learning, though it creates a mismatch between training conditions and real-world inference.