The Reference Model in DPO
What you'll learn: Why DPO needs a frozen reference model and how it keeps your policy from going off the rails.
The Role of the Reference Model
In Direct Preference Optimization (DPO), the reference model is a frozen copy of your starting policy (typically the SFT model). It never gets updated during training. Think of it as an anchor point—a reminder of where you began.
The DPO loss compares your current policy's log probabilities against this frozen reference. Specifically, it measures how much your policy has shifted from the reference on both preferred and dispreferred responses:
log π_θ(y_w|x) - log π_ref(y_w|x) [for preferred]
log π_θ(y_l|x) - log π_ref(y_l|x) [for dispreferred]
Why Keep It Frozen?
Without the reference model, your policy could overoptimize—it might learn to assign extreme probabilities to preferred responses while completely destroying its language modeling capabilities. You'd get a model that wins on your preference data but produces gibberish on everything else.
Analogy: Imagine learning to cook healthier meals. The reference model is your original cookbook. You want to improve recipes (optimize preferences), but you don't want to forget basic cooking techniques entirely. The frozen reference keeps you tethered to reasonable language patterns.
Preventing Mode Collapse
The reference model acts like the KL penalty term in RLHF, but it's implicit in DPO's formulation. By measuring relative log probability differences, DPO naturally penalizes large deviations. Your policy can improve on preferences while staying close to fluent, coherent language generation.
Key Takeaway: The frozen reference model in DPO prevents overoptimization by implicitly constraining how far your policy can drift from reasonable language behavior, all without needing a separate reward model or explicit KL penalty coefficient.