Why Word Embeddings: From One-Hot to Dense Vectors
What you'll learn: Why one-hot encoding fails to capture word meaning and how dense vectors solve this problem.
The Problem with One-Hot Encoding
When working with text, you need to convert words into numbers. The simplest approach is one-hot encoding: represent each word as a vector with all zeros except a single 1 at that word's position in the vocabulary.
For a vocabulary of 10,000 words, "cat" might be [0,0,1,0,0,...] and "dog" might be [0,1,0,0,0,...].
The fatal flaw: Every word is equally distant from every other word. "cat" and "dog" have no more relationship than "cat" and "spacecraft." The vectors are orthogonal — mathematically unrelated — even though cats and dogs are both animals.
Dense Vectors: Capturing Meaning
Word embeddings are dense, low-dimensional vectors (typically 50-300 dimensions) where similar words have similar vectors. Instead of 10,000 dimensions with one 1, you get 300 dimensions with real numbers like [0.2, -0.5, 0.8, ...].
These vectors are learned from data. Words appearing in similar contexts develop similar representations. Now "cat" and "dog" vectors point in similar directions because they share semantic properties.
Analogy: One-hot encoding is like a dictionary where each word gets its own isolated page. Word embeddings are like a 3D map where nearby locations represent related concepts — "cat" and "dog" cluster together in "animal" territory.
Key Takeaway: One-hot vectors treat all words as unrelated and scale poorly with vocabulary size. Dense word embeddings compress words into lower dimensions while encoding semantic relationships learned from text, enabling models to understand that similar words should behave similarly.