Course contentsShow
← Machine Learning and Deep Learning
Lesson 1781 of 3,53838. Instruction Tuning and AlignmentFree lesson

Preference Dataset Construction

Creating training pairs: sampling multiple completions per prompt and obtaining human rankings or comparisons.

Preference Dataset Construction

What you'll learn: How to build a dataset of ranked model outputs by generating multiple completions per prompt and collecting human judgments about which responses are better.

The Core Idea

In RLHF, your reward model needs examples of "good" vs "bad" responses to learn what humans prefer. But how do you create these training pairs? The answer is preference dataset construction: you take a single prompt, generate multiple different completions from your model, then ask humans to rank or compare them.

Think of it like a cooking competition. You give several chefs (model samples) the same recipe challenge (prompt), collect their different dishes (completions), then have judges (human annotators) taste and rank which they prefer. The judges don't need to explain why they prefer one dish — just which one tastes better overall.

The Construction Process

  1. Sample multiple completions: For each prompt in your dataset, generate 2-10 different responses using temperature sampling or other stochastic decoding methods
  2. Human annotation: Present pairs (or groups) of completions to human raters who select which response is better
  3. Format the data: Structure the results as (prompt, chosen_response, rejected_response) tuples

For example, given the prompt "Explain photosynthesis", you might generate five completions and ask annotators: "Which explanation is clearer and more accurate?" The winning response becomes "chosen," others become "rejected."

Why Multiple Samples Matter

Generating varied completions from the same prompt ensures your comparisons are meaningful. If responses are too similar, annotators can't distinguish preferences. If they're from different prompts, comparisons aren't valid.

Key Takeaway: Preference datasets are built by sampling multiple completions per prompt and collecting human rankings, creating the comparison pairs needed to train reward models that capture human preferences.