Course contentsShow
← Machine Learning and Deep Learning
Lesson 1186 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Left-to-Right vs Bidirectional Context

Comparing unidirectional (GPT-style) and bidirectional (BERT-style) context modeling and their trade-offs.

Left-to-Right vs Bidirectional Context

What you'll learn: How unidirectional (GPT-style) and bidirectional (BERT-style) models differ in how they read and understand text.

The Core Distinction

When language models process text, they can look at context in fundamentally different ways:

Left-to-Right (Unidirectional): Models like GPT read text exactly as you do when reading a book—one word at a time, from left to right. When predicting the next word, they only see what came before it. This is autoregressive modeling.

Bidirectional: Models like BERT read the entire sentence at once, looking both backward and forward around each word. When understanding "bank" in "I walked to the river bank," BERT sees both "river" (after) and "walked to" (before) simultaneously.

The Trade-offs

Unidirectional Advantages

  • Natural text generation: Perfect for writing, completing sentences, and chatbots because it predicts one word at a time
  • Straightforward training: The model simply learns to predict the next token given all previous tokens

Bidirectional Advantages

  • Richer understanding: Seeing full context in both directions helps with tasks like sentiment analysis, question answering, and classification
  • Better disambiguation: Words with multiple meanings are easier to understand with full context

The Cost

  • Bidirectional models can't generate text naturally because they'd see the future tokens they're supposed to predict
  • Unidirectional models miss backward context that could improve understanding

Key Takeaway: Left-to-right models excel at generation by predicting what comes next, while bidirectional models excel at understanding by seeing all context at once—each architecture is optimized for different types of NLP tasks.