Decoder-Only Design: Why GPT Diverged from BERT
You'll learn: Why GPT uses only the decoder stack instead of BERT's encoder-only or the full encoder-decoder architecture.
The Architectural Fork
When BERT was released, it used an encoder-only architecture with bidirectional attention—every token could see every other token. GPT took a fundamentally different path: it kept only the decoder stack from the original Transformer, using causal attention masking to prevent tokens from seeing future context.
Why Decoder-Only?
The decoder-only design aligns perfectly with autoregressive generation. Think of it like writing a story: you write one word at a time, building on what came before, never knowing what comes next until you create it. This is exactly what the decoder does—it predicts the next token based only on previous tokens.
BERT's bidirectional attention is powerful for understanding complete sentences (like reading a finished book), but it can't generate text naturally because it would "cheat" by seeing future words. GPT's decoder architecture enforces the constraint that generation must happen left-to-right, one token at a time.
Implications
This architectural choice means:
- Unified pretraining and generation: The same causal attention used during pretraining (next-token prediction) works seamlessly at inference
- Natural text generation: The model learns to generate coherent sequences without requiring a separate encoder
- Simpler architecture: No encoder-decoder attention mechanism needed
- Scalability: The single-stack design scales efficiently as you add parameters
The trade-off? Unlike BERT, GPT can't leverage bidirectional context during pretraining—each token only learns from its left-side neighbors.
Key Takeaway: GPT's decoder-only architecture with causal masking directly supports autoregressive generation, making it naturally suited for text generation tasks rather than bidirectional understanding tasks like BERT.