Course contentsShow
← Machine Learning and Deep Learning
Lesson 1777 of 3,53838. Instruction Tuning and AlignmentFree lesson

The Role of Reward Models in RLHF

How reward models convert human preferences into scalar signals for reinforcement learning optimization.

The Role of Reward Models in RLHF

You'll learn: How reward models translate human preferences into numerical signals that guide reinforcement learning optimization.

Why We Need Reward Models

In RLHF, we need to teach language models to produce outputs humans prefer—but reinforcement learning algorithms require scalar rewards (numbers), not vague human opinions. This is where reward models become the critical bridge.

Think of a reward model as a learned judge. Just as a gymnastics judge converts a performance into a score, the reward model converts a language model's output into a numerical value representing how well it aligns with human preferences.

How Reward Models Work

The reward model is trained on preference data—pairs of responses where humans indicated which one they liked better. During this training (using the Bradley-Terry model you've already learned), it learns to predict a scalar score for any model output.

Once trained, the reward model acts as a proxy for human judgment during RL fine-tuning. For every response the policy model generates, the reward model assigns it a score. Higher scores mean the output better aligns with human preferences.

The Critical Function

Without reward models, we'd need humans evaluating every single output during training—completely impractical. The reward model makes RLHF scalable by:

  • Converting qualitative preferences into quantitative signals
  • Providing consistent, instant feedback during RL optimization
  • Enabling thousands of training iterations without human intervention

The RL optimization then maximizes these reward scores (while the KL penalty prevents drift), gradually steering the model toward more aligned behavior.

Key Takeaway: Reward models are trained judges that convert human preferences into scalar reward signals, making it possible to optimize language models with reinforcement learning at scale.