Course contentsShow
← Machine Learning and Deep Learning
Lesson 1131 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Limitations of Static Word Embeddings

Why single vectors per word fail to capture polysemy, context-dependent meaning, and sentence-level semantics.

Limitations of Static Word Embeddings

You'll learn: Why assigning a single vector to each word fails to capture the richness and context-dependency of human language.

The Core Problem

Static word embeddings like Word2Vec and GloVe assign one fixed vector to each word, regardless of how that word is used. This seems efficient, but language is far more nuanced than that.

Why Single Vectors Fall Short

Polysemy: One Word, Many Meanings

The word "bank" has multiple meanings:

  • Financial institution: "I went to the bank to deposit money."
  • River edge: "We sat by the bank of the river."

Static embeddings give both uses the same vector, averaging across all contexts in the training data. The model can't distinguish which meaning applies.

Context-Dependent Meaning

Even beyond polysemy, words shift meaning subtly:

  • "The movie was sick" (awesome)
  • "I feel sick" (ill)

A single vector cannot capture these contextual nuances.

Sentence-Level Semantics Lost

Static embeddings treat sentences as bags of word vectors. They miss:

  • Word order: "Dog bites man" vs. "Man bites dog"
  • Syntax and grammar: Relationships between words
  • Negation: "not good" should mean something different than "good"

The Analogy

Imagine describing a person with one photograph taken years ago. That photo can't show whether they're happy today, what they're wearing now, or who they're with. Similarly, a static embedding is a "snapshot" that ignores the word's current context.

Key Takeaway: Static word embeddings fail because they assign one fixed representation per word, unable to capture polysemy, context-dependent meanings, or sentence-level relationships—motivating the need for contextual embeddings.