Course contentsShow
← Machine Learning and Deep Learning
Lesson 1765 of 3,53838. Instruction Tuning and AlignmentFree lesson

Preference Data Format and Structure

How to structure comparison data with prompts, chosen responses, and rejected responses for reward modeling.

Preference Data Format and Structure

What you'll learn: How to organize comparison data for training reward models in RLHF.

Core Concept

Once you've collected human feedback, you need to structure it properly for reward model training. Unlike supervised fine-tuning where you have single correct outputs, RLHF requires paired comparisons: for the same prompt, you need both a preferred (chosen) response and a less preferred (rejected) response.

Think of it like rating restaurant dishes. Instead of just saying "this pasta is 8/10," you're saying "for my dinner order, I preferred the pasta over the pizza." The prompt is your context (what you ordered for), and you're expressing a relative preference between two options.

The Standard Format

Each training example consists of three components:

  1. Prompt: The input instruction or query
  2. Chosen response: The output humans preferred or rated higher
  3. Rejected response: The output humans disliked or rated lower
{
  "prompt": "Explain photosynthesis simply",
  "chosen": "Plants use sunlight to convert CO₂ and water into glucose and oxygen.",
  "rejected": "Photosynthesis is complicated biochemical process involving multiple reactions..."
}

The reward model will learn to assign higher scores to chosen responses and lower scores to rejected responses given the same prompt. This teaches the model to distinguish better from worse outputs in a way that reflects human preferences.

Why This Structure Matters

Unlike classification labels (0 or 1), preferences are comparative. You're not saying one response is objectively good—you're saying it's better than the alternative for that specific prompt. This relative ranking is what makes reward modeling effective.

Key Takeaway: Preference data structures prompt-response pairs as triplets (prompt, chosen, rejected), enabling reward models to learn comparative quality judgments rather than absolute scores.