Retrieval Quality Metrics
What you'll learn: How to measure whether your RAG system finds and ranks the right documents using industry-standard metrics.
Why Measure Retrieval Quality?
When building RAG (Retrieval-Augmented Generation) systems, you need to know if your search component actually finds relevant information. Just like grading a search engine, you want to measure both what it retrieves and how well it ranks results.
Core Metrics Explained
Precision@k
Precision@k measures: "Of the top k results returned, how many were actually relevant?"
If you retrieve 5 documents and 3 are relevant, precision@5 = 3/5 = 0.6. It answers: "Can I trust what's at the top?"
Recall@k
Recall@k measures: "Of all relevant documents that exist, how many did the top k results capture?"
If 10 relevant docs exist and your top 5 captures 3 of them, recall@5 = 3/10 = 0.3. It answers: "Am I missing important information?"
MRR (Mean Reciprocal Rank)
MRR measures how quickly you hit the first relevant result. If the first relevant doc appears at position 3, the reciprocal rank = 1/3. Average this across queries to get MRR. Higher is better—users want answers fast.
NDCG (Normalized Discounted Cumulative Gain)
NDCG is sophisticated: it considers how relevant each result is (not just yes/no) and where it appears (position matters). Documents at position 1 count more than position 10. NDCG ranges 0-1, where 1.0 means perfect ranking.
Choosing the Right Metric
- Use precision@k when accuracy matters most (medical, legal)
- Use recall@k when completeness matters (research, investigation)
- Use MRR for single-answer questions (FAQ bots)
- Use NDCG when relevance has degrees and ranking quality matters
Key Takeaway: Retrieval quality metrics like precision@k, recall@k, MRR, and NDCG help you objectively measure whether your RAG system finds the right documents in the right order.