Creating Ground Truth Test Sets
You'll learn how to build reliable query-document relevance pairs that serve as gold standard data for measuring retrieval quality.
What Is a Ground Truth Test Set?
A ground truth test set is a curated collection of queries paired with known-correct documents that should be retrieved for each query. Think of it as an answer key for evaluating your retrieval system—just like a teacher needs an answer key to grade exams fairly, you need ground truth to measure whether your RAG pipeline retrieves the right information.
Each entry typically contains:
- A query (user question or search term)
- Relevant document IDs (which chunks/documents should be retrieved)
- Optionally, irrelevant document IDs (what should not appear)
Why Ground Truth Matters
Without ground truth, you're flying blind. Metrics like MRR, NDCG, and precision require you to know what the right answer is before you can calculate how well your system performs. Ground truth lets you:
- Compare different chunking strategies objectively
- Measure improvements from reranking or query expansion
- Catch regressions when changing embedding models
- Benchmark hybrid search versus dense retrieval
How to Build Ground Truth
- Manual annotation: Domain experts review real user queries and label which documents answer them
- Synthetic generation: Use LLMs to generate questions from documents (know the source = know the answer)
- User feedback: Collect clicks, ratings, or explicit relevance judgments from production
- Hybrid approach: Generate synthetically, then have humans validate
Start small—20-50 high-quality pairs beat 1000 noisy ones.
Key Takeaway: Ground truth test sets are labeled query-document pairs that provide objective standards for measuring retrieval performance, enabling you to calculate metrics like precision, recall, and NDCG with confidence.