Preference Dataset Construction
What you'll learn: How to build a dataset of ranked model outputs by generating multiple completions per prompt and collecting human judgments about which responses are better.
The Core Idea
In RLHF, your reward model needs examples of "good" vs "bad" responses to learn what humans prefer. But how do you create these training pairs? The answer is preference dataset construction: you take a single prompt, generate multiple different completions from your model, then ask humans to rank or compare them.
Think of it like a cooking competition. You give several chefs (model samples) the same recipe challenge (prompt), collect their different dishes (completions), then have judges (human annotators) taste and rank which they prefer. The judges don't need to explain why they prefer one dish — just which one tastes better overall.
The Construction Process
- Sample multiple completions: For each prompt in your dataset, generate 2-10 different responses using temperature sampling or other stochastic decoding methods
- Human annotation: Present pairs (or groups) of completions to human raters who select which response is better
- Format the data: Structure the results as (prompt, chosen_response, rejected_response) tuples
For example, given the prompt "Explain photosynthesis", you might generate five completions and ask annotators: "Which explanation is clearer and more accurate?" The winning response becomes "chosen," others become "rejected."
Why Multiple Samples Matter
Generating varied completions from the same prompt ensures your comparisons are meaningful. If responses are too similar, annotators can't distinguish preferences. If they're from different prompts, comparisons aren't valid.
Key Takeaway: Preference datasets are built by sampling multiple completions per prompt and collecting human rankings, creating the comparison pairs needed to train reward models that capture human preferences.