Course contentsShow
← Machine Learning and Deep Learning
Lesson 1117 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Why Word Embeddings: From One-Hot to Dense Vectors

Understand the limitations of one-hot encoding and why dense, learned representations capture semantic meaning better.

Why Word Embeddings: From One-Hot to Dense Vectors

What you'll learn: Why one-hot encoding fails to capture word meaning and how dense vectors solve this problem.

The Problem with One-Hot Encoding

When working with text, you need to convert words into numbers. The simplest approach is one-hot encoding: represent each word as a vector with all zeros except a single 1 at that word's position in the vocabulary.

For a vocabulary of 10,000 words, "cat" might be [0,0,1,0,0,...] and "dog" might be [0,1,0,0,0,...].

The fatal flaw: Every word is equally distant from every other word. "cat" and "dog" have no more relationship than "cat" and "spacecraft." The vectors are orthogonal — mathematically unrelated — even though cats and dogs are both animals.

Dense Vectors: Capturing Meaning

Word embeddings are dense, low-dimensional vectors (typically 50-300 dimensions) where similar words have similar vectors. Instead of 10,000 dimensions with one 1, you get 300 dimensions with real numbers like [0.2, -0.5, 0.8, ...].

These vectors are learned from data. Words appearing in similar contexts develop similar representations. Now "cat" and "dog" vectors point in similar directions because they share semantic properties.

Analogy: One-hot encoding is like a dictionary where each word gets its own isolated page. Word embeddings are like a 3D map where nearby locations represent related concepts — "cat" and "dog" cluster together in "animal" territory.

Key Takeaway: One-hot vectors treat all words as unrelated and scale poorly with vocabulary size. Dense word embeddings compress words into lower dimensions while encoding semantic relationships learned from text, enabling models to understand that similar words should behave similarly.