RL Fine-Tuning Setup: Policy and Reference Models
What You'll Learn
You'll understand how to set up the core model architecture for RL fine-tuning in RLHF, including the policy model that gets updated and the frozen reference model that prevents drift.
The Core Idea
After training your reward model, you need two versions of your SFT model for the RL fine-tuning phase:
-
Policy Model (Actor): This is your active model that generates responses and gets updated through reinforcement learning. It starts as a copy of your SFT model and learns to maximize the reward signal.
-
Reference Model: This is a frozen copy of the same SFT model that never gets updated. It serves as an anchor point to prevent your policy from drifting too far from the original instruction-following behavior.
Why Two Models?
Think of the reference model as your "safety tether." Without it, the policy could exploit the reward model by generating nonsensical but high-scoring outputs. The reference model keeps the policy grounded by penalizing outputs that diverge too much from the original SFT distribution.
The Actor-Critic Framework
This setup implements an actor-critic architecture:
- Actor = Policy Model: Takes actions (generates text)
- Critic = Reward Model: Evaluates how good those actions are
The policy generates text, receives rewards from the trained reward model, and uses those rewards (plus the KL penalty from comparing to the reference model) to update its parameters via reinforcement learning algorithms like PPO.
The reference model enables computing a KL divergence penalty that measures how different the policy's outputs are from the original model's distribution.
Key Takeaway: RL fine-tuning uses the SFT model twice—once as an updatable policy that learns from rewards, and once as a frozen reference that prevents the policy from forgetting its instruction-following abilities while optimizing for human preferences.