Course contentsShow
← Machine Learning and Deep Learning
Lesson 1152 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Bidirectional Context vs Autoregressive Models

Why BERT's bidirectional attention enables better understanding than left-to-right models like GPT.

Bidirectional Context vs Autoregressive Models

You'll learn: Why BERT's bidirectional attention mechanism provides deeper understanding than left-to-right models like GPT.

The Core Difference

Imagine reading a mystery novel in two different ways:

Autoregressive (like GPT): You read left-to-right, predicting the next word based only on what came before. When you reach "The detective solved the ___," you guess based on earlier clues, but you can't peek ahead.

Bidirectional (like BERT): You can read the entire sentence at once. For "The detective solved the _____ before midnight," you see both the context before AND after the blank, making your guess far more informed.

Why Bidirectionality Matters

Autoregressive models (GPT, traditional language models) use causal self-attention — they mask future tokens to prevent "cheating" during generation. This makes them excellent for text generation but limits their understanding. They're essentially playing a guessing game with one eye closed.

BERT's bidirectional attention sees the full sentence simultaneously. Every token attends to every other token in both directions. This deeper context understanding makes BERT superior for:

  • Question answering: Understanding how words relate across the entire question and passage
  • Sentiment analysis: Catching context clues that appear later in the sentence
  • Named entity recognition: Using surrounding context on both sides

The trade-off? BERT can't directly generate text (it doesn't know what comes "next"), but it excels at understanding and classification tasks.

Key Takeaway: BERT's bidirectional attention allows each token to see its full context simultaneously, creating richer representations than autoregressive models that only see leftward context — the difference between reading with full visibility versus reading through a narrow left-to-right window.