Preference Data Format and Structure
What you'll learn: How to organize comparison data for training reward models in RLHF.
Core Concept
Once you've collected human feedback, you need to structure it properly for reward model training. Unlike supervised fine-tuning where you have single correct outputs, RLHF requires paired comparisons: for the same prompt, you need both a preferred (chosen) response and a less preferred (rejected) response.
Think of it like rating restaurant dishes. Instead of just saying "this pasta is 8/10," you're saying "for my dinner order, I preferred the pasta over the pizza." The prompt is your context (what you ordered for), and you're expressing a relative preference between two options.
The Standard Format
Each training example consists of three components:
- Prompt: The input instruction or query
- Chosen response: The output humans preferred or rated higher
- Rejected response: The output humans disliked or rated lower
{
"prompt": "Explain photosynthesis simply",
"chosen": "Plants use sunlight to convert CO₂ and water into glucose and oxygen.",
"rejected": "Photosynthesis is complicated biochemical process involving multiple reactions..."
}
The reward model will learn to assign higher scores to chosen responses and lower scores to rejected responses given the same prompt. This teaches the model to distinguish better from worse outputs in a way that reflects human preferences.
Why This Structure Matters
Unlike classification labels (0 or 1), preferences are comparative. You're not saying one response is objectively good—you're saying it's better than the alternative for that specific prompt. This relative ranking is what makes reward modeling effective.
Key Takeaway: Preference data structures prompt-response pairs as triplets (prompt, chosen, rejected), enabling reward models to learn comparative quality judgments rather than absolute scores.