Course contentsShow
← Machine Learning and Deep Learning
Lesson 1132 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

The Contextualization Idea

How contextual embeddings assign different vectors to the same word based on surrounding context in a sentence.

The Contextualization Idea

What you'll learn: How contextual embeddings solve the problem of words having different meanings in different contexts by generating dynamic, sentence-specific vectors.

The Problem with Static Embeddings

Remember our static word embeddings like Word2Vec and GloVe? They assign one fixed vector to each word, regardless of where it appears. This creates a fundamental problem: many words have multiple meanings depending on context.

Consider the word "bank":

  • "I sat by the river bank" (landform)
  • "I deposited money at the bank" (financial institution)

With static embeddings, "bank" gets the same vector in both sentences, even though the meanings are completely different!

The Core Idea: Context-Dependent Vectors

Contextual embeddings solve this by generating a different vector for each occurrence of a word based on its surrounding context. The same word token gets transformed into different representations depending on the sentence it appears in.

Think of it like this: A static embedding is like a passport photo—the same image everywhere. A contextual embedding is like a video—capturing you differently in each moment based on what's happening around you.

How It Works (Conceptually)

Instead of looking up a fixed vector from a table, contextual models:

  1. Start with an initial representation of the word
  2. Look at the surrounding words in the sentence
  3. Use mechanisms like attention to understand the context
  4. Generate a new, context-aware vector for that specific occurrence

The word "bank" near "river" produces a different vector than "bank" near "money"—even though they started from the same initial representation.

Key Takeaway: Contextual embeddings assign different vectors to the same word based on surrounding context, allowing models to capture polysemy (multiple meanings) and disambiguate words dynamically within each sentence.