Course contentsShow
← Machine Learning and Deep Learning
Lesson 1807 of 3,53838. Instruction Tuning and AlignmentFree lesson

DPO Loss: Mathematical Formulation

Understanding the log-sigmoid loss over preferred vs rejected completion probabilities.

DPO Loss: Mathematical Formulation

You'll learn: How DPO translates preference comparisons into a concrete training objective using log-sigmoid loss over completion probability ratios.

The Core Mathematical Expression

Direct Preference Optimization skips the reward model entirely by optimizing the language model directly on preference data. The DPO loss function operationalizes this idea mathematically.

Given a preferred completion y_w ("winner") and a rejected completion y_l ("loser") for prompt x, DPO computes:

L_DPO = -log σ(β · log[π_θ(y_w|x) / π_ref(y_w|x)] - β · log[π_θ(y_l|x) / π_ref(y_l|x)])

Where:

  • π_θ is your policy model (the model you're training)
  • π_ref is the frozen reference model (usually the SFT model)
  • β is a temperature parameter controlling deviation from reference
  • σ is the sigmoid function

Understanding the Components

The log probability ratio log[π_θ(y) / π_ref(y)] measures how much more (or less) likely your current policy makes a completion compared to the reference.

The difference between ratios compares how the policy treats the preferred vs rejected completions. You want preferred completions to have higher ratios than rejected ones.

The log-sigmoid converts this difference into a probability that mirrors the Bradley-Terry preference model, creating a smooth loss that:

  • Approaches 0 when preferred completions are strongly favored
  • Increases when rejected completions become competitive

Think of it as asking: "Does my model prefer the winner more strongly than the loser, relative to the reference model?"

Key Takeaway: The DPO loss uses log-sigmoid over implicit reward differences (encoded as probability ratios) to directly optimize preference alignment without training a separate reward model.