Encoder Output Reuse
What you'll learn: How the encoder processes input just once while the decoder uses that same output repeatedly across all generation steps.
The Core Efficiency Insight
In an encoder-decoder architecture, the encoder runs a single forward pass through the entire input sequence and produces its output (the encoded representations). This encoder output is then reused by the decoder at every single step of autoregressive generation.
Think of it like reading a book once to understand it thoroughly, then answering multiple questions about it without re-reading. The encoder "reads" the source sequence once, creating a rich representation. The decoder then "consults" this same representation at each generation step through cross-attention—asking different questions but always referring back to the same encoded understanding.
Why This Matters
During inference, if you're generating a 50-token translation, the encoder runs exactly once on the input sentence, but the decoder runs 50 times (one for each output token). At each decoder step:
- The decoder performs masked self-attention on tokens generated so far
- The decoder performs cross-attention using those same encoder outputs
- A new token is predicted
The encoder outputs remain frozen and unchanged throughout this entire process. This is computationally efficient—you don't re-encode the source repeatedly—and it's why caching encoder outputs is a common optimization strategy.
Practical Impact
This reuse pattern means that once you've encoded "The cat sat on the mat" into its representation vectors, every word you generate in the translation ("Le", "chat", "était"...) attends back to that identical encoded representation. The encoder's job is done after one pass.
Key Takeaway: The encoder processes the input sequence once and produces outputs that are reused unchanged by the decoder across all autoregressive generation steps, making the architecture efficient for sequence-to-sequence tasks.