Course contentsShow
← Machine Learning and Deep Learning
Lesson 1215 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Encoder-Decoder vs Decoder-Only Architectures

Compare architectural differences between encoder-decoder models and GPT-style decoder-only models for different tasks.

Encoder-Decoder vs Decoder-Only Architectures

What you'll learn: You'll understand the fundamental architectural differences between encoder-decoder models (like T5) and decoder-only models (like GPT), and when each shines.

The Core Difference

Think of the encoder-decoder architecture as a translator with two specialists: one who deeply understands the input (encoder using bidirectional attention) and another who generates the output (decoder using causal attention). The encoder reads the entire input at once, builds rich representations, then passes them to the decoder.

In contrast, decoder-only models like GPT are single-track specialists: they only use causal (left-to-right) attention throughout. There's no separate encoding phase—the model processes everything autoregressively, predicting one token at a time based only on what came before.

When Each Architecture Excels

Encoder-Decoder models shine at tasks requiring deep input understanding before generation:

  • Translation (read full French sentence → generate English)
  • Summarization (comprehend entire document → produce summary)
  • Question answering with context passages

Decoder-Only models excel at:

  • Open-ended text generation
  • In-context learning (examples become part of the input sequence)
  • Simplicity and scalability (one architecture for all tasks)
  • Conversational AI like ChatGPT

The Trade-Off

Encoder-decoder models process input bidirectionally, capturing richer context from both directions. However, they're more complex with two separate components.

Decoder-only models are simpler and scale better to massive sizes (like GPT-3's 175B parameters), but they can only see previous tokens, not future ones. Yet GPT models compensate through sheer scale and next-token prediction training on enormous datasets.

Key Takeaway: Encoder-decoder architectures use separate bidirectional encoding and autoregressive decoding for structured tasks, while decoder-only models use purely causal attention throughout—simpler, more scalable, and dominant in modern large language models.