Reward Model Architecture and Training Objective
What you'll learn: How to turn a pretrained language model into a "judge" that scores which responses humans prefer.
The Core Idea
After collecting human preference data (comparing response pairs), we need a model that can predict which response humans would prefer. The reward model (RM) does exactly this: it takes the same base model architecture you used for supervised fine-tuning, removes its language modeling head, and adds a scalar head — a simple layer that outputs a single number representing how "good" a response is.
Architecture Details
Start with your SFT model's weights (the instruction-tuned checkpoint). Replace the vocabulary projection layer with a linear layer that outputs one number. When you feed in a prompt + response, the reward model processes it like normal, but instead of predicting the next token, it outputs a single reward score.
Pairwise Ranking Loss
Here's where it gets clever: you don't train on absolute "goodness" scores. Instead, you train on comparisons. Given two responses to the same prompt — one preferred (winner) and one rejected (loser) — you want:
reward(prompt + winner) > reward(prompt + loser)
The typical loss function is:
loss = -log(sigmoid(reward_winner - reward_loser))
This pushes the model to assign higher scores to preferred responses while maintaining relative rankings. It's like teaching a judge to compare apples — you show pairs and say "this one's better," not "this one is exactly 7.3 good."
Key Takeaway: The reward model reuses your base architecture with a scalar output head, trained via pairwise ranking loss to predict human preferences by learning that winning responses should score higher than rejected ones.