Course contentsShow
← Machine Learning and Deep Learning
Lesson 1128 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Limitations of Static Embeddings

Recognize the fundamental limitation: each word has one fixed representation regardless of context or meaning.

Limitations of Static Embeddings

What You'll Learn

You'll understand why traditional word embeddings like Word2Vec and GloVe have a critical flaw: they assign one fixed vector per word, regardless of how that word is used.

The One-Representation Problem

Static embeddings (Word2Vec, GloVe) create a single, unchanging vector for each word in your vocabulary. This means "bank" gets the exact same representation whether you're talking about a river bank or a financial institution. The word "apple" has identical embeddings when discussing fruit or technology companies.

Why This Matters

Think of it like having one photograph to represent a person in every context of their life—at work, at home, playing sports, or sleeping. That single image can't capture all the different roles and contexts where that person appears.

In natural language:

  • Polysemy: Words have multiple meanings ("bat" = animal or sports equipment)
  • Context-dependent usage: "The movie was sick" vs "I feel sick" use the same embedding despite opposite sentiments
  • Named entities: "Paris" (city) vs "Paris" (person's name) are indistinguishable

The Technical Reality

When you load Word2Vec or GloVe embeddings, you get a lookup table: each word maps to one fixed 300-dimensional vector (or whatever dimension you chose). No matter what sentence you place that word in, it retrieves the same vector. The embedding has no awareness of surrounding words, grammar, or semantic context.

This limitation motivated the development of contextual embeddings where representations change based on usage—but that's for future lessons!

Key Takeaway: Static embeddings give every word exactly one fixed representation, making them unable to distinguish between different meanings, contexts, or uses of the same word—a fundamental limitation for understanding natural language.