Course contentsShow
← Machine Learning and Deep Learning
Lesson 1126 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Evaluating Word Embeddings: Intrinsic Methods

Learn intrinsic evaluation through word similarity tasks, analogy tasks, and correlation with human judgments.

Evaluating Word Embeddings: Intrinsic Methods

You'll learn how to measure the quality of word embeddings without building a full downstream application.

The Core Idea

Once you've trained word embeddings (like Word2Vec or GloVe), how do you know if they're actually good? Intrinsic evaluation tests embeddings directly on specific linguistic tasks, without needing a complete NLP system. Think of it like testing a car engine on a dynamometer before installing it in a vehicle—you measure performance in isolation.

Three Main Intrinsic Methods

1. Word Similarity Tasks

Compare embedding-based similarity with human judgments. Datasets like WordSim-353 contain word pairs (e.g., "cat" and "dog") with human-assigned similarity scores. You compute cosine similarity between embeddings and measure correlation with human ratings using Spearman's or Pearson's correlation coefficient.

2. Word Analogy Tasks

Test if embeddings capture semantic relationships. The classic example: "king - man + woman ≈ queen". You compute embedding(king) - embedding(man) + embedding(woman) and check if the nearest vector is embedding(queen). Accuracy across many analogies (capital cities, verb tenses, gender) indicates quality.

3. Correlation with Human Judgments

Beyond similarity, this includes categorical tasks like identifying which word doesn't belong in a group (e.g., "breakfast, cereal, dinner, lunch" → "cereal" is different). Embeddings should cluster conceptually related words together.

Why This Matters

Intrinsic evaluation is fast and interpretable—you immediately know if your embeddings capture basic linguistic properties. However, remember that high intrinsic scores don't guarantee good performance on real tasks like sentiment analysis or translation.

Key Takeaway: Intrinsic evaluation methods—word similarity, analogy tasks, and correlation with human judgments—provide quick, focused metrics to assess whether embeddings capture meaningful semantic relationships before using them in applications.