Inference Efficiency: Encoder-Decoder vs Decoder-Only
What You'll Learn
You'll understand how computational costs and memory requirements differ between encoder-decoder and decoder-only models during inference.
The Core Difference
When generating text, encoder-decoder models (like T5 and BART) process input once through the encoder, then generate output token-by-token through the decoder. Decoder-only models (like GPT) treat everything as a sequence and generate autoregressively from start to finish.
Computational Cost Comparison
Encoder-Decoder advantages:
- The encoder processes the input sequence just once at the start
- Only the decoder runs autoregressively during generation
- If your input is long and output is short, you save computation
Decoder-Only characteristics:
- Must reprocess the entire context (input + generated tokens) at each generation step
- As output grows, each new token becomes more expensive to compute
- The attention mechanism looks back at all previous tokens every time
Memory Requirements
Encoder-decoder models store:
- Encoder representations (fixed size, computed once)
- Decoder's key-value cache for generated tokens
Decoder-only models store:
- Key-value cache for the entire sequence (input + output combined)
- This cache grows with both input length and generation length
The Practical Impact
Think of it like cooking: encoder-decoder models prep all ingredients once (encode), then cook step-by-step (decode). Decoder-only models re-examine the entire recipe and all previous steps before each new action. For tasks with short outputs, encoder-decoder can be more efficient. For long generations or when the distinction between "input" and "output" is blurry, decoder-only simplicity often wins.
Key Takeaway: Encoder-decoder models are more inference-efficient when inputs are long and outputs are short, while decoder-only models have simpler architecture but higher computational cost that scales with total sequence length.