The Distributional Hypothesis
What you'll learn: You'll understand the foundational principle that words appearing in similar contexts tend to have similar meanings—the bedrock of modern word embeddings.
The Core Idea
The distributional hypothesis is elegantly simple: you shall know a word by the company it keeps. Words that appear in similar contexts likely share similar meanings. For example, if "cat" and "dog" both frequently appear near words like "pet," "furry," "feed," and "veterinarian," we can infer they're semantically related.
This insight, dating back to linguist J.R. Firth in the 1950s, revolutionized how we represent words in machine learning. Instead of treating words as arbitrary symbols (like one-hot vectors that say nothing about meaning), we can learn representations that capture semantic similarity by analyzing where and with what words appear.
Why This Matters for Word Embeddings
Before word embeddings, we used sparse one-hot vectors where "king" and "queen" were equally different from each other as "king" and "potato." The distributional hypothesis gives us a path forward: by examining millions of text examples, we can automatically discover that "king" and "queen" appear in remarkably similar contexts (throne, crown, reign, royal), and therefore should have similar vector representations.
This principle underpins techniques like Word2Vec and GloVe, which construct dense vectors where geometric distance reflects semantic similarity—all derived from contextual patterns in text.
A Simple Analogy
Think of words as people and contexts as social gatherings. If Alice and Bob consistently attend the same types of events (tech conferences, coffee shops, book clubs), you'd reasonably guess they share similar interests. The distributional hypothesis applies this same logic to language.
Key Takeaway: The distributional hypothesis—words in similar contexts have similar meanings—transforms word representation from arbitrary symbols into meaningful vectors learned from statistical patterns in text.