Evaluating Search Quality
What you'll learn: How to measure whether your semantic search actually returns the right results using precision, recall, and human judgments.
Why Evaluation Matters
Building a semantic search system is one thing—knowing if it works well is another. Just because you can compute cosine similarity and retrieve top-K results doesn't mean users are finding what they need. Evaluation turns your intuition into measurable facts.
The Core Metrics
Precision asks: "Of the results I returned, how many were actually relevant?" If your search returns 10 documents and 7 are useful, that's 70% precision.
Recall asks: "Of all the relevant documents that exist, how many did I find?" If there are 20 relevant documents total in your collection and you retrieved 7 of them, that's 35% recall.
Think of precision as accuracy (no junk results) and recall as completeness (didn't miss the good stuff). There's often a trade-off: lowering your similarity threshold might increase recall but hurt precision.
Manual Relevance Judgments
Someone—usually a human—must decide what "relevant" means for each query. You create a test set: specific queries paired with lists of which documents should be considered relevant. This is your ground truth.
For example: Query "python error handling" → Documents [5, 12, 23] are relevant. Then run your search and compare what you retrieved against this gold standard.
Key Takeaway: Precision measures how accurate your results are, recall measures how complete they are, and both require human-labeled test data to calculate meaningfully. Without evaluation, you're flying blind.