Training the Reward Model: Data Requirements
What you'll learn: How to determine the right amount and quality of preference data needed to train an effective reward model for RLHF.
Core Idea
Training a reward model to predict human preferences requires careful consideration of data quantity, quality, and consistency. Unlike supervised fine-tuning where you need thousands of examples, reward modeling can work with surprisingly fewer comparisons—but the quality bar is much higher.
How Much Data Do You Need?
For a typical reward model, you can start with 10,000 to 50,000 preference pairs. This is much less than the millions of tokens needed for pretraining. The key is diversity: your data should cover the range of outputs your model will generate during RL training.
Think of it like training a judge for a cooking competition. You don't need to taste 10,000 dishes—but you do need to sample enough variety (appetizers, mains, desserts) to develop consistent judgment criteria.
Data Quality Considerations
Quality trumps quantity in reward modeling:
- Clear preferences: Avoid comparisons where both outputs are equally good/bad
- Task coverage: Include examples spanning your use cases (helpfulness, safety, formatting)
- Output diversity: Ensure both chosen and rejected responses vary in quality dimensions
Poor data leads to a reward model that misunderstands what "better" means, causing RL to optimize the wrong objective.
Handling Annotator Disagreement
Human annotators often disagree—and that's informative! Options include:
- Majority vote: Use the preference most annotators chose
- Agreement filtering: Drop examples with <70% agreement
- Model uncertainty: Train the reward model to express confidence on controversial examples
High disagreement may signal ambiguous instructions or genuinely subjective preferences, both requiring attention.
Key Takeaway: Reward model training succeeds with tens of thousands of high-quality, diverse preference pairs; prioritize clear comparisons and address annotator disagreement systematically to avoid misaligned objectives.