Course contentsShow
← Machine Learning and Deep Learning
Lesson 1226 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Inference Efficiency: Encoder-Decoder vs Decoder-Only

Compare computational costs and memory requirements of encoder-decoder models versus decoder-only alternatives at inference time.

Inference Efficiency: Encoder-Decoder vs Decoder-Only

What You'll Learn

You'll understand how computational costs and memory requirements differ between encoder-decoder and decoder-only models during inference.

The Core Difference

When generating text, encoder-decoder models (like T5 and BART) process input once through the encoder, then generate output token-by-token through the decoder. Decoder-only models (like GPT) treat everything as a sequence and generate autoregressively from start to finish.

Computational Cost Comparison

Encoder-Decoder advantages:

  • The encoder processes the input sequence just once at the start
  • Only the decoder runs autoregressively during generation
  • If your input is long and output is short, you save computation

Decoder-Only characteristics:

  • Must reprocess the entire context (input + generated tokens) at each generation step
  • As output grows, each new token becomes more expensive to compute
  • The attention mechanism looks back at all previous tokens every time

Memory Requirements

Encoder-decoder models store:

  • Encoder representations (fixed size, computed once)
  • Decoder's key-value cache for generated tokens

Decoder-only models store:

  • Key-value cache for the entire sequence (input + output combined)
  • This cache grows with both input length and generation length

The Practical Impact

Think of it like cooking: encoder-decoder models prep all ingredients once (encode), then cook step-by-step (decode). Decoder-only models re-examine the entire recipe and all previous steps before each new action. For tasks with short outputs, encoder-decoder can be more efficient. For long generations or when the distinction between "input" and "output" is blurry, decoder-only simplicity often wins.

Key Takeaway: Encoder-decoder models are more inference-efficient when inputs are long and outputs are short, while decoder-only models have simpler architecture but higher computational cost that scales with total sequence length.