Course contentsShow
← Machine Learning and Deep Learning
Lesson 1981 of 3,53842. Retrieval-Augmented Generation: FoundationsFree lesson

Embedding Model Evaluation Metrics

Learn metrics like MRR, NDCG, and Recall@K to properly evaluate retrieval model performance on benchmark datasets.

Embedding Model Evaluation Metrics

You'll learn how to measure whether your retrieval embeddings actually find the right documents using industry-standard metrics.

Why Evaluation Metrics Matter

When you train or select an embedding model for retrieval (like those Sentence Transformers you've been learning about), you need objective ways to measure performance. Is your bi-encoder finding the right documents? How good is that cross-encoder reranker? Without metrics, you're flying blind.

Core Retrieval Metrics

Mean Reciprocal Rank (MRR)

MRR measures where the first relevant document appears in your ranked results. If the correct document is rank 1, you get 1.0. If it's rank 2, you get 0.5. If rank 3, you get 0.33. Average these reciprocal ranks across all queries.

Analogy: You're asking for the nearest coffee shop. MRR rewards systems that put the right answer at the very top.

Normalized Discounted Cumulative Gain (NDCG)

NDCG considers all relevant documents in your results, not just the first. Documents ranked higher contribute more to the score, with diminishing returns further down the list. Scores are normalized to [0, 1], where 1 means perfect ranking.

Use case: When multiple documents could answer the query well, NDCG captures overall ranking quality better than MRR.

Recall@K

Recall@K asks: "Of all relevant documents, what percentage did I find in my top K results?" If there are 10 relevant docs and you found 7 in your top-20 results, Recall@20 = 0.7.

Trade-off: Higher K usually means better recall but more noise for your LLM to process.

Practical Application

You evaluate on benchmark datasets with labeled query-document pairs (like those used in contrastive training). Run your retrieval pipeline, compute these metrics, then iterate on your embedding model or retrieval strategy.

Key Takeaway: MRR measures first-hit quality, NDCG evaluates full ranking quality, and Recall@K measures coverage—use all three to fully understand your retrieval system's strengths and weaknesses.