Course contentsShow
← AI Engineering
Lesson 236 of 1,8866. Embeddings and Semantic SearchFree lesson

Evaluating Search Quality

Measuring semantic search performance using precision, recall, and manual relevance judgments.

Evaluating Search Quality

What you'll learn: How to measure whether your semantic search actually returns the right results using precision, recall, and human judgments.

Why Evaluation Matters

Building a semantic search system is one thing—knowing if it works well is another. Just because you can compute cosine similarity and retrieve top-K results doesn't mean users are finding what they need. Evaluation turns your intuition into measurable facts.

The Core Metrics

Precision asks: "Of the results I returned, how many were actually relevant?" If your search returns 10 documents and 7 are useful, that's 70% precision.

Recall asks: "Of all the relevant documents that exist, how many did I find?" If there are 20 relevant documents total in your collection and you retrieved 7 of them, that's 35% recall.

Think of precision as accuracy (no junk results) and recall as completeness (didn't miss the good stuff). There's often a trade-off: lowering your similarity threshold might increase recall but hurt precision.

Manual Relevance Judgments

Someone—usually a human—must decide what "relevant" means for each query. You create a test set: specific queries paired with lists of which documents should be considered relevant. This is your ground truth.

For example: Query "python error handling" → Documents [5, 12, 23] are relevant. Then run your search and compare what you retrieved against this gold standard.

Key Takeaway: Precision measures how accurate your results are, recall measures how complete they are, and both require human-labeled test data to calculate meaningfully. Without evaluation, you're flying blind.