Course contentsShow
← Machine Learning and Deep Learning
Lesson 1030 of 3,53823. Recurrent Neural NetworksFree lesson

Inference and Autoregressive Generation

How seq2seq models generate outputs at test time by feeding predicted tokens back as inputs iteratively.

Inference and Autoregressive Generation

What you'll learn: How sequence-to-sequence models generate outputs one token at a time during inference by feeding each prediction back as the next input.

The Core Idea

During training, you used teacher forcing — feeding the model the correct previous tokens at each step. But at test time, you don't have those correct tokens yet! The model must generate the sequence step-by-step, using its own predictions.

This is called autoregressive generation: each output becomes the next input, creating a chain of predictions that builds the complete sequence.

How It Works

  1. Start with the encoder output: Your encoder processes the input sequence and produces the context vector (the "bottleneck" representation)
  2. Initialize the decoder: Feed a special <START> token as the first input
  3. Generate token-by-token: The decoder predicts the most likely next token
  4. Feed back: That predicted token becomes the input for the next decoding step
  5. Repeat: Continue until the model predicts an <END> token or reaches a maximum length

Think of it like writing a story where you can only see the sentence you just wrote — each new word depends on what you've already generated, not on what the "correct" story should be.

The Key Difference

  • Training (teacher forcing): Decoder sees [<START>, "Hello", "world"] → predicts ["Hello", "world", <END>]
  • Inference (autoregressive): Decoder sees [<START>] → predicts "Hello" → sees [<START>, "Hello"] → predicts "world" → continues...

This sequential dependency means inference is inherently slower than training, since you can't parallelize the token generation.

Key Takeaway: Autoregressive generation produces outputs iteratively by feeding each predicted token back as input, creating a self-dependent chain where errors can accumulate but the model generates complete sequences autonomously.