Course contentsShow
← Machine Learning and Deep Learning
Lesson 1335 of 3,53829. Core NLP TasksFree lesson

Evaluating Semantic Search Systems

Measure performance with Mean Reciprocal Rank, Recall@k, and nDCG on standard retrieval benchmarks like BEIR.

Evaluating Semantic Search Systems

What You'll Learn

You'll learn how to measure whether your semantic search system actually returns relevant results using industry-standard metrics.

Why Evaluation Matters

After building a semantic search system with sentence transformers and dense embeddings, you need objective ways to measure its quality. Unlike text classification where answers are clear-cut, search relevance is nuanced—does the 3rd result matter as much as the 1st? Should you care about top-5 or top-100?

Core Metrics

Mean Reciprocal Rank (MRR) answers: "How high up is the first relevant result?" If the correct answer appears at position 3, that query gets score 1/3. Average across all queries for final MRR. Think of it as rewarding systems that put at least one right answer near the top.

Recall@k asks: "Of all relevant documents, what percentage appear in my top-k results?" If 5 documents are relevant and 3 appear in your top-10, Recall@10 = 0.6. It measures coverage, not ranking quality.

nDCG (Normalized Discounted Cumulative Gain) is the most sophisticated metric. It considers both position and relevance levels (not just binary relevant/irrelevant). Results at position 1 count more than position 10, and "highly relevant" documents score better than "somewhat relevant" ones. The "normalized" part lets you compare across queries.

BEIR Benchmark

BEIR (Benchmarking IR) provides standard datasets across diverse domains—science papers, questions, fact-checking—letting you test if your model generalizes beyond its training distribution. It's become the go-to for evaluating semantic search robustness.

Key Takeaway: Evaluate semantic search with MRR (first relevant hit), Recall@k (coverage), and nDCG (graded relevance + position); BEIR offers standardized cross-domain benchmarks to prove your system works beyond familiar data.