Course contentsShow
← Machine Learning and Deep Learning
Lesson 1099 of 3,53824. The Transformer ArchitectureFree lesson

Training with Teacher Forcing

Parallel training by feeding ground-truth tokens to decoder with appropriate masking.

Training with Teacher Forcing

What You'll Learn

You'll learn how teacher forcing enables efficient parallel training of Transformer decoder models by feeding ground-truth tokens instead of predictions during training.

The Core Problem

When training encoder-decoder models like Transformers for sequence generation, the decoder needs to predict one token at a time. If we waited for each prediction before computing the next one during training, it would be extremely slow and sequential—losing the parallelization advantages that make Transformers powerful.

How Teacher Forcing Works

Teacher forcing solves this by feeding the actual ground-truth tokens from the target sequence into the decoder during training, rather than the model's own predictions. This allows the decoder to compute all output positions simultaneously in a single forward pass.

Here's the clever part: we use masked self-attention in the decoder to prevent each position from "seeing" future tokens. Even though all ground-truth tokens are available at once, the mask ensures position i can only attend to positions 1 through i, preserving the autoregressive property.

An Analogy

Imagine learning to translate sentences. With teacher forcing, you're shown the correct translation word-by-word as you practice, even when you make mistakes. You don't have to wait for your answer before seeing the next word. The masking ensures you're still learning the sequential dependencies—you can't peek ahead at words you haven't "generated" yet.

Training vs Inference

During training: use ground-truth tokens with masked attention (parallel). During inference: use model's own predictions sequentially (autoregressive generation).

This mismatch is intentional—it dramatically speeds up training while the mask maintains proper causality.

Key Takeaway: Teacher forcing enables parallel training of autoregressive decoders by feeding ground-truth tokens with causal masking, making Transformer training far more efficient than sequential generation.