Encoder-Decoder vs Decoder-Only Architectures
What you'll learn: You'll understand the fundamental architectural differences between encoder-decoder models (like T5) and decoder-only models (like GPT), and when each shines.
The Core Difference
Think of the encoder-decoder architecture as a translator with two specialists: one who deeply understands the input (encoder using bidirectional attention) and another who generates the output (decoder using causal attention). The encoder reads the entire input at once, builds rich representations, then passes them to the decoder.
In contrast, decoder-only models like GPT are single-track specialists: they only use causal (left-to-right) attention throughout. There's no separate encoding phase—the model processes everything autoregressively, predicting one token at a time based only on what came before.
When Each Architecture Excels
Encoder-Decoder models shine at tasks requiring deep input understanding before generation:
- Translation (read full French sentence → generate English)
- Summarization (comprehend entire document → produce summary)
- Question answering with context passages
Decoder-Only models excel at:
- Open-ended text generation
- In-context learning (examples become part of the input sequence)
- Simplicity and scalability (one architecture for all tasks)
- Conversational AI like ChatGPT
The Trade-Off
Encoder-decoder models process input bidirectionally, capturing richer context from both directions. However, they're more complex with two separate components.
Decoder-only models are simpler and scale better to massive sizes (like GPT-3's 175B parameters), but they can only see previous tokens, not future ones. Yet GPT models compensate through sheer scale and next-token prediction training on enormous datasets.
Key Takeaway: Encoder-decoder architectures use separate bidirectional encoding and autoregressive decoding for structured tasks, while decoder-only models use purely causal attention throughout—simpler, more scalable, and dominant in modern large language models.