Course contentsShow
← AI Engineering
Lesson 210 of 1,8866. Embeddings and Semantic SearchFree lesson

Contextual vs Static Embeddings

Compare context-dependent transformer embeddings (BERT) with static word embeddings (Word2Vec, GloVe).

Contextual vs Static Embeddings

What you'll learn: How embeddings can either stay the same for a word or change based on surrounding context.

The Core Difference

Until now, we've talked about embeddings as if each word or token always maps to the same vector. That's how static embeddings like Word2Vec and GloVe work—the word "bank" always gets the exact same vector, whether you're talking about a river bank or a financial bank.

Contextual embeddings (like those from BERT and modern transformers) generate different vectors for the same word depending on the sentence it appears in. The model reads the whole sentence and adjusts each word's embedding based on its neighbors.

A Simple Analogy

Think of static embeddings like a dictionary definition: "bank" has one fixed meaning you look up every time.

Contextual embeddings are like reading the word in a full sentence—you understand "I sat by the river bank" differently than "I went to the bank for a loan" because you see the context around it.

Why This Matters

Static embeddings struggle with:

  • Polysemy: Words with multiple meanings
  • Context-dependent nuances: "good" in "good food" vs "good enough"

Contextual embeddings capture:

  • Meaning in context: Same word, different vectors for different uses
  • Relationships to nearby words: The embedding changes based on what's around it

Modern AI systems (BERT, GPT, etc.) use contextual embeddings because they're dramatically better at understanding what words actually mean in each specific situation.

Key Takeaway: Static embeddings give the same vector to a word every time; contextual embeddings generate different vectors based on the surrounding sentence, capturing meaning more accurately.