Course contentsShow
← Machine Learning and Deep Learning
Lesson 1770 of 3,53838. Instruction Tuning and AlignmentFree lesson

RL Fine-Tuning Setup: Policy and Reference Models

Using the SFT model as initial policy, keeping a frozen reference model, and the actor-critic framework.

RL Fine-Tuning Setup: Policy and Reference Models

What You'll Learn

You'll understand how to set up the core model architecture for RL fine-tuning in RLHF, including the policy model that gets updated and the frozen reference model that prevents drift.

The Core Idea

After training your reward model, you need two versions of your SFT model for the RL fine-tuning phase:

  1. Policy Model (Actor): This is your active model that generates responses and gets updated through reinforcement learning. It starts as a copy of your SFT model and learns to maximize the reward signal.

  2. Reference Model: This is a frozen copy of the same SFT model that never gets updated. It serves as an anchor point to prevent your policy from drifting too far from the original instruction-following behavior.

Why Two Models?

Think of the reference model as your "safety tether." Without it, the policy could exploit the reward model by generating nonsensical but high-scoring outputs. The reference model keeps the policy grounded by penalizing outputs that diverge too much from the original SFT distribution.

The Actor-Critic Framework

This setup implements an actor-critic architecture:

  • Actor = Policy Model: Takes actions (generates text)
  • Critic = Reward Model: Evaluates how good those actions are

The policy generates text, receives rewards from the trained reward model, and uses those rewards (plus the KL penalty from comparing to the reference model) to update its parameters via reinforcement learning algorithms like PPO.

The reference model enables computing a KL divergence penalty that measures how different the policy's outputs are from the original model's distribution.

Key Takeaway: RL fine-tuning uses the SFT model twice—once as an updatable policy that learns from rewards, and once as a frozen reference that prevents the policy from forgetting its instruction-following abilities while optimizing for human preferences.