Course contentsShow
← Machine Learning and Deep Learning
Lesson 1785 of 3,53838. Instruction Tuning and AlignmentFree lesson

Evaluating Reward Model Quality

Metrics for reward models: accuracy on held-out preferences, agreement with human judgments, and correlation analysis.

Evaluating Reward Model Quality

What you'll learn: How to measure whether your reward model actually captures human preferences accurately and reliably.

Why Evaluation Matters

You've trained a reward model to predict human preferences—but how do you know if it's any good? A poorly calibrated reward model will mislead your RL fine-tuning, causing the policy to optimize for the wrong behaviors. Rigorous evaluation prevents you from building on a faulty foundation.

Core Metrics for Reward Model Quality

Accuracy on Held-Out Preferences

The most straightforward metric: how often does your reward model correctly predict which response humans preferred? Split your preference dataset into train/validation/test sets, then measure classification accuracy. If your model says response A scores 0.8 and response B scores 0.6, does that match the actual human preference for A over B?

Think of this like a taste test judge: if they correctly predict which dish the panel preferred 85% of the time, they're a reliable judge.

Agreement with Human Judgments

Beyond binary accuracy, measure how closely your model's preference rankings align with human annotators. Use metrics like agreement rate across multiple comparisons or Kendall's tau for ranking correlation. This reveals whether the model's relative scoring matches human intuition on a broader scale.

Correlation Analysis

Examine how your model's raw scores correlate with various human judgment dimensions: helpfulness, harmfulness, truthfulness. Strong correlations suggest your model captures the underlying preferences; weak or negative correlations signal misalignment.

Also check inter-annotator agreement: if humans disagree heavily on certain examples, your model shouldn't be penalized for "wrong" predictions on inherently ambiguous cases.

Key Takeaway: Evaluate reward models with held-out preference accuracy, agreement rates with human rankings, and correlation analyses to ensure they reliably capture human values before using them for RL fine-tuning.