Creating Ground Truth for Retrieval
What you'll learn: How to build high-quality relevance judgments that serve as the foundation for evaluating and improving your RAG system's retrieval performance.
The Core Challenge
You've learned metrics like NDCG, MAP, and MRR—but they all require ground truth: labeled data that tells you which documents are actually relevant to each query. Without ground truth, you can't measure retrieval quality or train better models.
Creating ground truth means building a dataset of query-document pairs with relevance labels. Think of it like having answer keys for a test: you need these "correct answers" to grade your retrieval system.
Three Primary Approaches
Manual Annotation
Human experts review query-document pairs and assign relevance scores (e.g., 0-3 scale: not relevant, somewhat relevant, relevant, highly relevant). This is the gold standard but expensive and time-consuming. Best for critical domains where accuracy matters most.
LLM Generation
Use a powerful LLM to generate queries that a document should answer, or to judge relevance between existing query-document pairs. This scales much better than human annotation but may inherit the LLM's biases. Works well for initial ground truth that you later validate with humans.
Mining Existing QA Pairs
Extract ground truth from existing question-answer datasets, support tickets, or FAQs. If you have "Question → Answer" pairs, the documents containing those answers are relevant to those questions. This is fast and domain-specific, but limited to what already exists.
Practical Considerations
Start small: 50-100 high-quality annotated queries beats 1000 noisy ones. Mix approaches—use LLM generation for scale, then validate critical examples manually. Include both positive examples (relevant docs) and hard negatives (similar but not relevant) for robust evaluation.
Key Takeaway: Ground truth is the foundation of retrieval evaluation; build it through human annotation for quality, LLM generation for scale, or mining existing QA pairs for efficiency—often combining all three approaches.