Course contentsShow
← Machine Learning and Deep Learning
Lesson 2029 of 3,53843. Retrieval-Augmented Generation: Advanced TechniquesFree lesson

Creating Ground Truth for Retrieval

Building relevance judgments through annotation, LLM generation, or extracting from existing QA pairs.

Creating Ground Truth for Retrieval

What you'll learn: How to build high-quality relevance judgments that serve as the foundation for evaluating and improving your RAG system's retrieval performance.

The Core Challenge

You've learned metrics like NDCG, MAP, and MRR—but they all require ground truth: labeled data that tells you which documents are actually relevant to each query. Without ground truth, you can't measure retrieval quality or train better models.

Creating ground truth means building a dataset of query-document pairs with relevance labels. Think of it like having answer keys for a test: you need these "correct answers" to grade your retrieval system.

Three Primary Approaches

Manual Annotation

Human experts review query-document pairs and assign relevance scores (e.g., 0-3 scale: not relevant, somewhat relevant, relevant, highly relevant). This is the gold standard but expensive and time-consuming. Best for critical domains where accuracy matters most.

LLM Generation

Use a powerful LLM to generate queries that a document should answer, or to judge relevance between existing query-document pairs. This scales much better than human annotation but may inherit the LLM's biases. Works well for initial ground truth that you later validate with humans.

Mining Existing QA Pairs

Extract ground truth from existing question-answer datasets, support tickets, or FAQs. If you have "Question → Answer" pairs, the documents containing those answers are relevant to those questions. This is fast and domain-specific, but limited to what already exists.

Practical Considerations

Start small: 50-100 high-quality annotated queries beats 1000 noisy ones. Mix approaches—use LLM generation for scale, then validate critical examples manually. Include both positive examples (relevant docs) and hard negatives (similar but not relevant) for robust evaluation.

Key Takeaway: Ground truth is the foundation of retrieval evaluation; build it through human annotation for quality, LLM generation for scale, or mining existing QA pairs for efficiency—often combining all three approaches.