Preference Data Collection
What you'll learn: How to gather human judgments comparing different model outputs—the foundation of training reward models in RLHF.
The Core Idea
Preference data collection is the process of asking humans to compare different model responses and indicate which one is better. Instead of providing absolute ratings or rewriting responses, annotators make comparative judgments: "Output A is better than Output B for this prompt."
This approach works because humans find it easier to compare options than to rate them independently. Just as you might struggle to rate a restaurant "7.3 out of 10" but can easily say "this meal was better than last week's," preference comparisons are more consistent and reliable.
Three Main Collection Methods
Ranking Responses
Present annotators with multiple responses (3-5 typically) for the same prompt and ask them to rank from best to worst. This produces rich preference data: if you have outputs A, B, C ranked in that order, you extract pairs: A>B, A>C, B>C.
Pairwise Preferences
Show two responses side-by-side for a single prompt and ask "Which is better?" This is the most common approach because it's cognitively simple and produces clean binary data. You can later aggregate many pairwise comparisons.
Rating Scales
Ask annotators to rate each response independently (e.g., 1-5 stars for helpfulness, harmlessness, accuracy). While less commonly used in modern RLHF, ratings can be converted to preferences: a response rated 5 is preferred over one rated 3.
Why This Matters for RLHF
The Bradley-Terry model (which you've learned) transforms these preference pairs into reward model training data. Quality preference collection directly determines how well your reward model captures human values.
Key Takeaway: Preference data collection gathers human comparison judgments through ranking, pairwise selection, or ratings—creating the training signal that teaches reward models to distinguish better from worse model outputs.