Course contentsShow
← Machine Learning and Deep Learning
Lesson 1806 of 3,53838. Instruction Tuning and AlignmentFree lesson

Deriving the DPO Loss Function

From the Bradley-Terry model and optimal RLHF policy to the closed-form DPO objective.

Deriving the DPO Loss Function

What you'll learn: How DPO transforms the RLHF optimization problem into a simple classification loss by reparameterizing the optimal policy.

From RLHF to DPO

In RLHF, we learned the optimal policy satisfies:

π*(y|x) ∝ π_ref(y|x) · exp(r(x,y) / β)

where r is the reward, π_ref is the reference model, and β controls KL divergence. The problem? We need to train a separate reward model and run expensive RL (PPO).

The Key Insight

DPO asks: What if we solve for the reward function in terms of the policy? Rearranging the RLHF optimal policy equation:

r(x,y) = β · log(π*(y|x) / π_ref(y|x)) + constant

Now substitute this into the Bradley-Terry preference model. For preferred completion y_w over y_l:

P(y_w ≻ y_l) = σ(r(x,y_w) - r(x,y_l))

Replace both rewards with our policy-based expression:

P(y_w ≻ y_l) = σ(β · log(π(y_w|x)/π_ref(y_w|x)) - β · log(π(y_l|x)/π_ref(y_l|x)))

The DPO Loss

This gives us the closed-form DPO objective:

L_DPO = -𝔼[(log σ(β · log(π_θ(y_w|x)/π_ref(y_w|x)) - β · log(π_θ(y_l|x)/π_ref(y_l|x))))]

Instead of training a reward model and running RL, we directly optimize the policy to prefer y_w over y_l using preference pairs—it's just binary classification with log-probabilities!

Why This Matters

DPO eliminates the reward model and PPO entirely. You train your language model directly on preference data as if it were a classification problem, implicitly maximizing reward and maintaining KL constraint.

Key Takeaway: DPO derives a single-stage loss by analytically solving for the reward in terms of policy ratios, converting RLHF's multi-stage pipeline into direct preference probability maximization under the Bradley-Terry model.