Course contentsShow
← Machine Learning and Deep Learning
Lesson 1188 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Teacher Forcing in Autoregressive Training

Using ground-truth tokens as input at each step during training to enable parallel computation of losses.

Teacher Forcing in Autoregressive Training

What You'll Learn

You'll learn how teacher forcing accelerates autoregressive model training by feeding ground-truth tokens instead of model predictions during each step.

The Core Idea

When training an autoregressive language model (like GPT), you need the model to predict the next token given all previous tokens. But here's the challenge: if you waited for the model to generate token 2 before predicting token 3, training would be painfully slow—you'd have to process sequences one token at a time.

Teacher forcing solves this by "cheating" during training: instead of feeding the model's own predictions as input for the next step, we feed it the actual ground-truth tokens from our training data.

How It Works

Imagine teaching someone to complete the sentence "The cat sat on the ___".

  • Without teacher forcing: Student guesses "mat", then you ask them to continue from "The cat sat on the mat..." (using their own guess)
  • With teacher forcing: Student guesses "mat", but you show them the correct answer was "rug", and ask them to continue from "The cat sat on the rug..." (using the truth)

In practice, this means during training on the sentence "Hello world", the model:

  • Sees [H] → predicts e
  • Sees [H, e] → predicts l
  • Sees [H, e, l] → predicts l

All these predictions happen in parallel because we know all the ground-truth inputs ahead of time. The causal attention mask (from lesson 1187) ensures each position only attends to previous positions, maintaining the autoregressive property.

Why It Matters

Teacher forcing enables parallel loss computation across all tokens in a sequence simultaneously, making training hundreds of times faster than sequential generation would be.

Key Takeaway: Teacher forcing uses ground-truth tokens as input during training, allowing parallel computation of all next-token predictions at once rather than waiting for sequential generation.