Course contentsShow
← Machine Learning and Deep Learning
Lesson 1767 of 3,53838. Instruction Tuning and AlignmentFree lesson

Reward Model Architecture and Training Objective

Using the base model with a scalar head to predict preference scores and the pairwise ranking loss.

Reward Model Architecture and Training Objective

What you'll learn: How to turn a pretrained language model into a "judge" that scores which responses humans prefer.

The Core Idea

After collecting human preference data (comparing response pairs), we need a model that can predict which response humans would prefer. The reward model (RM) does exactly this: it takes the same base model architecture you used for supervised fine-tuning, removes its language modeling head, and adds a scalar head — a simple layer that outputs a single number representing how "good" a response is.

Architecture Details

Start with your SFT model's weights (the instruction-tuned checkpoint). Replace the vocabulary projection layer with a linear layer that outputs one number. When you feed in a prompt + response, the reward model processes it like normal, but instead of predicting the next token, it outputs a single reward score.

Pairwise Ranking Loss

Here's where it gets clever: you don't train on absolute "goodness" scores. Instead, you train on comparisons. Given two responses to the same prompt — one preferred (winner) and one rejected (loser) — you want:

reward(prompt + winner) > reward(prompt + loser)

The typical loss function is:

loss = -log(sigmoid(reward_winner - reward_loser))

This pushes the model to assign higher scores to preferred responses while maintaining relative rankings. It's like teaching a judge to compare apples — you show pairs and say "this one's better," not "this one is exactly 7.3 good."

Key Takeaway: The reward model reuses your base architecture with a scalar output head, trained via pairwise ranking loss to predict human preferences by learning that winning responses should score higher than rejected ones.