Training with Teacher Forcing
What You'll Learn
You'll learn how teacher forcing enables efficient parallel training of Transformer decoder models by feeding ground-truth tokens instead of predictions during training.
The Core Problem
When training encoder-decoder models like Transformers for sequence generation, the decoder needs to predict one token at a time. If we waited for each prediction before computing the next one during training, it would be extremely slow and sequential—losing the parallelization advantages that make Transformers powerful.
How Teacher Forcing Works
Teacher forcing solves this by feeding the actual ground-truth tokens from the target sequence into the decoder during training, rather than the model's own predictions. This allows the decoder to compute all output positions simultaneously in a single forward pass.
Here's the clever part: we use masked self-attention in the decoder to prevent each position from "seeing" future tokens. Even though all ground-truth tokens are available at once, the mask ensures position i can only attend to positions 1 through i, preserving the autoregressive property.
An Analogy
Imagine learning to translate sentences. With teacher forcing, you're shown the correct translation word-by-word as you practice, even when you make mistakes. You don't have to wait for your answer before seeing the next word. The masking ensures you're still learning the sequential dependencies—you can't peek ahead at words you haven't "generated" yet.
Training vs Inference
During training: use ground-truth tokens with masked attention (parallel). During inference: use model's own predictions sequentially (autoregressive generation).
This mismatch is intentional—it dramatically speeds up training while the mask maintains proper causality.
Key Takeaway: Teacher forcing enables parallel training of autoregressive decoders by feeding ground-truth tokens with causal masking, making Transformer training far more efficient than sequential generation.