Course contentsShow
← Machine Learning and Deep Learning
Lesson 1145 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

BERT's Encoder-Only Transformer Architecture

Why BERT uses only the encoder stack and how it differs from decoder-only or encoder-decoder models.

BERT's Encoder-Only Transformer Architecture

What you'll learn: Why BERT uses only encoder layers and how this design choice fundamentally differs from other Transformer variants.

The Encoder-Only Design Decision

BERT (Bidirectional Encoder Representations from Transformers) uses only the encoder stack from the original Transformer architecture—no decoder at all. This design choice directly supports BERT's core purpose: understanding language bidirectionally to create rich contextual representations.

Why Only Encoders?

Think of the encoder as a "comprehension module" that reads the entire input at once, while the decoder is a "generation module" that produces output one token at a time. BERT's job isn't to generate text—it's to understand it deeply.

The encoder stack allows:

  • Bidirectional attention: Every token can attend to every other token simultaneously (no masking required in self-attention)
  • Full context awareness: Each word sees both left and right neighbors at once
  • Parallel processing: All positions processed together, not sequentially

How This Differs from Other Architectures

Decoder-only models (like GPT) use causal masking—tokens only see previous context. They're designed for generation.

Encoder-decoder models (like the original Transformer for translation) have separate comprehension and generation modules connected by cross-attention. They're built for sequence-to-sequence tasks.

BERT (encoder-only) sacrifices generation capability to maximize bidirectional understanding. It's optimized for tasks where you need to understand meaning: classification, named entity recognition, question answering.

The encoder-only architecture makes BERT perfect for its Masked Language Modeling objective—it needs to see context on both sides of a masked token to predict it accurately.

Key Takeaway: BERT uses only encoders because it's designed for understanding language bidirectionally, not generating it—this architectural choice enables full bidirectional attention and makes BERT ideal for comprehension tasks rather than generation.