Retrieval Quality Metrics for RAG
You'll learn how to measure whether your RAG system is finding and using the right documents to answer user questions.
When building Retrieval-Augmented Generation (RAG) systems, retrieving relevant context is half the battle. Even the best language model will fail if you feed it irrelevant documents. Just like a researcher judging whether they found the right sources for their paper, you need metrics to evaluate your retrieval quality.
Core Retrieval Metrics
Precision measures what percentage of retrieved documents are actually relevant. If you fetch 10 documents and only 6 are useful, your precision is 60%. High precision means you're not wasting context window space on noise.
Recall measures what percentage of all relevant documents you successfully retrieved. If 8 relevant documents exist but you only fetched 5, your recall is 62.5%. High recall ensures you're not missing critical information.
Mean Reciprocal Rank (MRR) focuses on the position of the first relevant document. If the most relevant doc is ranked #3, you get 1/3 = 0.33. This matters because earlier-ranked documents often influence the LLM more.
Normalized Discounted Cumulative Gain (NDCG) is more sophisticated—it considers both relevance scores and ranking positions, heavily weighting top results. Think of it like judging search results where position matters exponentially.
Context Relevance
Beyond individual document metrics, context relevance scores measure how well your entire retrieved context set matches the query's intent. You can compute this using embedding similarity or even ask an LLM to judge relevance.
These metrics help you tune retrieval parameters (top-k, similarity thresholds, reranking), compare different embedding models, and detect when your RAG system degrades over time.
Key Takeaway: Track precision, recall, MRR, NDCG, and context relevance to ensure your RAG pipeline retrieves the right documents in the right order—poor retrieval quality undermines even the best generation models.