DPO Loss: Mathematical Formulation
You'll learn: How DPO translates preference comparisons into a concrete training objective using log-sigmoid loss over completion probability ratios.
The Core Mathematical Expression
Direct Preference Optimization skips the reward model entirely by optimizing the language model directly on preference data. The DPO loss function operationalizes this idea mathematically.
Given a preferred completion y_w ("winner") and a rejected completion y_l ("loser") for prompt x, DPO computes:
L_DPO = -log σ(β · log[π_θ(y_w|x) / π_ref(y_w|x)] - β · log[π_θ(y_l|x) / π_ref(y_l|x)])
Where:
π_θis your policy model (the model you're training)π_refis the frozen reference model (usually the SFT model)βis a temperature parameter controlling deviation from referenceσis the sigmoid function
Understanding the Components
The log probability ratio log[π_θ(y) / π_ref(y)] measures how much more (or less) likely your current policy makes a completion compared to the reference.
The difference between ratios compares how the policy treats the preferred vs rejected completions. You want preferred completions to have higher ratios than rejected ones.
The log-sigmoid converts this difference into a probability that mirrors the Bradley-Terry preference model, creating a smooth loss that:
- Approaches 0 when preferred completions are strongly favored
- Increases when rejected completions become competitive
Think of it as asking: "Does my model prefer the winner more strongly than the loser, relative to the reference model?"
Key Takeaway: The DPO loss uses log-sigmoid over implicit reward differences (encoded as probability ratios) to directly optimize preference alignment without training a separate reward model.