Course contentsShow
← Machine Learning and Deep Learning
Lesson 1137 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

BERT Embeddings as Contextual Representations

How BERT's bidirectional Transformer encoder produces context-dependent embeddings for each token in a sentence.

BERT Embeddings as Contextual Representations

You'll learn how BERT creates embeddings that change based on surrounding context, unlike static word embeddings.

The Core Problem with Static Embeddings

Remember Word2Vec and GloVe? They give the word "bank" the exact same vector whether you're talking about a river bank or a financial bank. That's a fundamental limitation we've been building toward solving.

How BERT Solves This

BERT (Bidirectional Encoder Representations from Transformers) uses a bidirectional Transformer encoder to produce embeddings that depend on the entire sentence context.

Here's how it works:

1. Bidirectional Processing

Unlike ELMo's separate left-to-right and right-to-left LSTMs, BERT's self-attention mechanism looks at all words simultaneously in both directions. When processing "bank" in "I deposited money at the bank," BERT's attention layers examine "deposited" and "money" on the left and can see the full context at once.

2. Token-Level Context Integration

Each token passes through multiple Transformer encoder layers. At each layer:

  • Self-attention computes relationships with every other token
  • The representation gets refined based on these relationships
  • By the final layer, each token's embedding captures its meaning in this specific sentence

3. Different Sentences, Different Embeddings

The same word produces different embeddings in different contexts:

  • "The bank was steep" → embedding influenced by "steep"
  • "The bank approved the loan" → embedding influenced by "approved" and "loan"

The Analogy

Think of static embeddings as a dictionary definition—always the same. BERT embeddings are like understanding a word through the conversation it appears in—the meaning shifts based on what's being discussed.

Key Takeaway: BERT's bidirectional Transformer encoder produces context-dependent embeddings by using self-attention to integrate information from all surrounding tokens, giving each word a unique representation for each sentence.