Evaluating Word Embeddings: Intrinsic Methods
You'll learn how to measure the quality of word embeddings without building a full downstream application.
The Core Idea
Once you've trained word embeddings (like Word2Vec or GloVe), how do you know if they're actually good? Intrinsic evaluation tests embeddings directly on specific linguistic tasks, without needing a complete NLP system. Think of it like testing a car engine on a dynamometer before installing it in a vehicle—you measure performance in isolation.
Three Main Intrinsic Methods
1. Word Similarity Tasks
Compare embedding-based similarity with human judgments. Datasets like WordSim-353 contain word pairs (e.g., "cat" and "dog") with human-assigned similarity scores. You compute cosine similarity between embeddings and measure correlation with human ratings using Spearman's or Pearson's correlation coefficient.
2. Word Analogy Tasks
Test if embeddings capture semantic relationships. The classic example: "king - man + woman ≈ queen". You compute embedding(king) - embedding(man) + embedding(woman) and check if the nearest vector is embedding(queen). Accuracy across many analogies (capital cities, verb tenses, gender) indicates quality.
3. Correlation with Human Judgments
Beyond similarity, this includes categorical tasks like identifying which word doesn't belong in a group (e.g., "breakfast, cereal, dinner, lunch" → "cereal" is different). Embeddings should cluster conceptually related words together.
Why This Matters
Intrinsic evaluation is fast and interpretable—you immediately know if your embeddings capture basic linguistic properties. However, remember that high intrinsic scores don't guarantee good performance on real tasks like sentiment analysis or translation.
Key Takeaway: Intrinsic evaluation methods—word similarity, analogy tasks, and correlation with human judgments—provide quick, focused metrics to assess whether embeddings capture meaningful semantic relationships before using them in applications.