Deriving the DPO Loss Function
What you'll learn: How DPO transforms the RLHF optimization problem into a simple classification loss by reparameterizing the optimal policy.
From RLHF to DPO
In RLHF, we learned the optimal policy satisfies:
π*(y|x) ∝ π_ref(y|x) · exp(r(x,y) / β)
where r is the reward, π_ref is the reference model, and β controls KL divergence. The problem? We need to train a separate reward model and run expensive RL (PPO).
The Key Insight
DPO asks: What if we solve for the reward function in terms of the policy? Rearranging the RLHF optimal policy equation:
r(x,y) = β · log(π*(y|x) / π_ref(y|x)) + constant
Now substitute this into the Bradley-Terry preference model. For preferred completion y_w over y_l:
P(y_w ≻ y_l) = σ(r(x,y_w) - r(x,y_l))
Replace both rewards with our policy-based expression:
P(y_w ≻ y_l) = σ(β · log(π(y_w|x)/π_ref(y_w|x)) - β · log(π(y_l|x)/π_ref(y_l|x)))
The DPO Loss
This gives us the closed-form DPO objective:
L_DPO = -𝔼[(log σ(β · log(π_θ(y_w|x)/π_ref(y_w|x)) - β · log(π_θ(y_l|x)/π_ref(y_l|x))))]
Instead of training a reward model and running RL, we directly optimize the policy to prefer y_w over y_l using preference pairs—it's just binary classification with log-probabilities!
Why This Matters
DPO eliminates the reward model and PPO entirely. You train your language model directly on preference data as if it were a classification problem, implicitly maximizing reward and maintaining KL constraint.
Key Takeaway: DPO derives a single-stage loss by analytically solving for the reward in terms of policy ratios, converting RLHF's multi-stage pipeline into direct preference probability maximization under the Bradley-Terry model.