Inference and Autoregressive Generation
What you'll learn: How sequence-to-sequence models generate outputs one token at a time during inference by feeding each prediction back as the next input.
The Core Idea
During training, you used teacher forcing — feeding the model the correct previous tokens at each step. But at test time, you don't have those correct tokens yet! The model must generate the sequence step-by-step, using its own predictions.
This is called autoregressive generation: each output becomes the next input, creating a chain of predictions that builds the complete sequence.
How It Works
- Start with the encoder output: Your encoder processes the input sequence and produces the context vector (the "bottleneck" representation)
- Initialize the decoder: Feed a special
<START>token as the first input - Generate token-by-token: The decoder predicts the most likely next token
- Feed back: That predicted token becomes the input for the next decoding step
- Repeat: Continue until the model predicts an
<END>token or reaches a maximum length
Think of it like writing a story where you can only see the sentence you just wrote — each new word depends on what you've already generated, not on what the "correct" story should be.
The Key Difference
- Training (teacher forcing): Decoder sees
[<START>, "Hello", "world"]→ predicts["Hello", "world", <END>] - Inference (autoregressive): Decoder sees
[<START>]→ predicts"Hello"→ sees[<START>, "Hello"]→ predicts"world"→ continues...
This sequential dependency means inference is inherently slower than training, since you can't parallelize the token generation.
Key Takeaway: Autoregressive generation produces outputs iteratively by feeding each predicted token back as input, creating a self-dependent chain where errors can accumulate but the model generates complete sequences autonomously.