Reward Model Architecture
What you'll learn: How to transform a pretrained language model into a reward model by adding a scalar output head.
Core Idea
A reward model takes text as input and outputs a single number (a scalar) representing how "good" or "aligned" that text is. Since we already have powerful pretrained LLMs that understand language, we don't build reward models from scratch—we adapt them.
The architecture is surprisingly simple: take your pretrained LLM (like the SFT model from earlier RLHF stages), keep all the transformer layers frozen or fine-tunable, but replace the language modeling head with a new component that produces a single scalar value instead of a probability distribution over vocabulary tokens.
The Modification Process
Think of it like repurposing a thermometer. Your pretrained LLM is like a detailed weather station that outputs temperature, humidity, pressure, and wind speed. For the reward model, you only need one reading—a single "quality score"—so you replace the complex multi-output display with a simple single-number gauge.
Technically, here's what happens:
- Keep the backbone: All transformer layers remain (they encode the input text into rich representations)
- Remove the LM head: Strip away the layer that predicts next tokens (typically a large linear layer projecting to vocabulary size)
- Add a scalar head: Replace it with a small linear layer that projects the final hidden state down to a single number—the reward
The output is just one float value per input sequence, representing how well that text aligns with human preferences.
Key Takeaway: Reward models reuse pretrained LLM architectures but swap the token-prediction head for a scalar output layer, converting a language generator into a quality evaluator.