Course contentsShow
← Machine Learning and Deep Learning
Lesson 1769 of 3,53838. Instruction Tuning and AlignmentFree lesson

Training the Reward Model: Data Requirements

How much preference data is needed, data quality considerations, and handling annotator disagreement.

Training the Reward Model: Data Requirements

What you'll learn: How to determine the right amount and quality of preference data needed to train an effective reward model for RLHF.

Core Idea

Training a reward model to predict human preferences requires careful consideration of data quantity, quality, and consistency. Unlike supervised fine-tuning where you need thousands of examples, reward modeling can work with surprisingly fewer comparisons—but the quality bar is much higher.

How Much Data Do You Need?

For a typical reward model, you can start with 10,000 to 50,000 preference pairs. This is much less than the millions of tokens needed for pretraining. The key is diversity: your data should cover the range of outputs your model will generate during RL training.

Think of it like training a judge for a cooking competition. You don't need to taste 10,000 dishes—but you do need to sample enough variety (appetizers, mains, desserts) to develop consistent judgment criteria.

Data Quality Considerations

Quality trumps quantity in reward modeling:

  • Clear preferences: Avoid comparisons where both outputs are equally good/bad
  • Task coverage: Include examples spanning your use cases (helpfulness, safety, formatting)
  • Output diversity: Ensure both chosen and rejected responses vary in quality dimensions

Poor data leads to a reward model that misunderstands what "better" means, causing RL to optimize the wrong objective.

Handling Annotator Disagreement

Human annotators often disagree—and that's informative! Options include:

  • Majority vote: Use the preference most annotators chose
  • Agreement filtering: Drop examples with <70% agreement
  • Model uncertainty: Train the reward model to express confidence on controversial examples

High disagreement may signal ambiguous instructions or genuinely subjective preferences, both requiring attention.

Key Takeaway: Reward model training succeeds with tens of thousands of high-quality, diverse preference pairs; prioritize clear comparisons and address annotator disagreement systematically to avoid misaligned objectives.