Training Data for Retrieval Models
You'll learn: How retrieval models like Sentence Transformers learn to distinguish relevant from irrelevant passages through carefully constructed positive and negative training pairs.
The Core Idea
Retrieval models trained with contrastive learning (as you learned earlier) need to see examples of what matches and what doesn't. The training data consists of positive pairs (query + relevant document) and negative pairs (query + irrelevant document). The quality and diversity of these pairs directly determines how well your retrieval system performs.
Sources of Training Pairs
Question-Answer Datasets
Datasets like MS MARCO or Natural Questions provide queries with known relevant passages. The question becomes the query, the passage containing the answer is a positive, and random other passages serve as negatives (called "random negatives").
Clickthrough Data
Real user behavior is gold. When someone searches "best Italian restaurants NYC" and clicks on a result, that's an implicit positive pair. Pages they skipped? Weak negatives. This captures actual information needs.
Synthetic Generation
You can create training data by:
- Generating questions from documents using an LLM
- Paraphrasing existing queries for data augmentation
- Creating "hard negatives" by retrieving top-K results with BM25, then marking non-answers as challenging negatives
Why Negatives Matter
Random negatives are too easy—the model learns surface patterns. Hard negatives (passages that look relevant but aren't) force the model to learn semantic understanding. Think of it like studying: multiple-choice questions with obviously wrong answers teach you less than ones with plausible distractors.
Key Takeaway: Retrieval models require positive pairs (query + relevant doc) and especially hard negative pairs (query + plausible-but-wrong doc) from sources like QA datasets, user clicks, or synthetic generation to learn meaningful semantic similarity.