Course contentsShow
← Machine Learning and Deep Learning
Lesson 1787 of 3,53838. Instruction Tuning and AlignmentFree lesson

Reward Model Data Quality

Impact of annotator agreement, label noise, and strategies for improving preference dataset quality and consistency.

Reward Model Data Quality

What you'll learn: How the quality and consistency of human preference data directly impacts reward model reliability and RLHF outcomes.

Why Data Quality Matters

A reward model is only as good as the preference data it's trained on. If human annotators disagree about which response is better, or if labels contain noise, your reward model learns inconsistent patterns. Think of it like learning to judge cooking from chefs who constantly contradict each other—you'd never develop a reliable sense of what "good" means.

Annotator Agreement

Inter-annotator agreement measures how often different annotators make the same preference choice for the same prompt-response pairs. Low agreement (say, 60%) means your "ground truth" is actually quite ambiguous. This introduces label noise that confuses the reward model during training.

High-quality datasets typically aim for 80%+ agreement on clear-cut cases, though some subjective preferences naturally have lower consensus.

Sources of Label Noise

  • Subjective preferences: What's "helpful" or "harmless" can vary by person
  • Annotator fatigue: Quality drops over long labeling sessions
  • Ambiguous instructions: Vague annotation guidelines create inconsistency
  • Edge cases: Truly close comparisons where either response is acceptable

Strategies for Improving Quality

  1. Clear guidelines: Provide detailed rubrics with examples
  2. Multi-annotator voting: Collect 3+ labels per pair and use majority vote
  3. Agreement filtering: Remove pairs with low annotator consensus
  4. Calibration sessions: Train annotators together on sample data
  5. Quality audits: Regularly review annotator work and provide feedback
  6. Tie handling: Allow annotators to mark genuinely equal responses

Key Takeaway: High-quality preference data with strong annotator agreement is essential for training reliable reward models—invest in clear guidelines, multiple annotations, and filtering low-agreement examples to reduce label noise.