Course contentsShow
← Machine Learning and Deep Learning
Lesson 1141 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Comparing Contextual Embedding Approaches

Differences between bidirectional (BERT), unidirectional (GPT), and hybrid contextual representations.

Comparing Contextual Embedding Approaches

What you'll learn: How bidirectional (BERT), unidirectional (GPT), and hybrid contextual embedding methods differ in how they capture word meaning.

The Core Difference: Direction of Context

When creating contextual embeddings, the fundamental choice is which surrounding words influence a token's representation.

Bidirectional (BERT-style)

BERT reads text in both directions simultaneously. When encoding the word "bank" in "I deposited money at the bank," BERT sees all words—both before ("deposited money") and after (if any). This gives the richest context but requires seeing the complete sequence upfront.

Analogy: Reading a sentence with both eyes open, seeing everything at once.

Unidirectional (GPT-style)

GPT reads text left-to-right only (causal/autoregressive). When encoding "bank," it only sees "I deposited money at the" but not what comes after. This is necessary for text generation (you can't see the future you're creating) but captures less context for understanding.

Analogy: Reading word-by-word with a sliding window that only reveals what you've already read.

Hybrid Approaches

Some models combine both:

  • ELMo trains separate forward and backward LSTMs, then concatenates their representations
  • Some architectures use bidirectional encoders with unidirectional decoders

When to Use Which

  • BERT (bidirectional): Best for understanding tasks (classification, NER, QA) where you have the full input
  • GPT (unidirectional): Required for generation tasks; also works for understanding by treating it as completion
  • Hybrid (ELMo): Bridges both worlds but less powerful than transformer-based approaches

Key Takeaway: Bidirectional embeddings (BERT) see full context and excel at understanding; unidirectional embeddings (GPT) only see past context, enabling generation but potentially missing future clues; hybrid methods attempt to capture both directions separately.