Implementing DPO in PyTorch
You'll learn how to build a complete DPO training loop in PyTorch, from loading model pairs to computing the preference loss and updating weights.
Core Idea
Direct Preference Optimization requires three components: a policy model (being trained), a reference model (frozen copy), and paired preference data. Your training loop computes log probabilities for chosen and rejected completions under both models, calculates the DPO loss using the beta hyperparameter, and updates only the policy model.
Code Walkthrough
Loading Model Pairs
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load policy model (trainable)
policy_model = AutoModelForCausalLM.from_pretrained("model_name")
policy_model.train()
# Load reference model (frozen)
reference_model = AutoModelForCausalLM.from_pretrained("model_name")
reference_model.eval()
for param in reference_model.parameters():
param.requires_grad = False
Computing DPO Loss
import torch
import torch.nn.functional as F
def dpo_loss(policy_chosen_logps, policy_rejected_logps,
reference_chosen_logps, reference_rejected_logps, beta=0.1):
"""Compute DPO loss from log probabilities"""
policy_logratios = policy_chosen_logps - policy_rejected_logps
reference_logratios = reference_chosen_logps - reference_rejected_logps
losses = -F.logsigmoid(beta * (policy_logratios - reference_logratios))
return losses.mean()
Training Loop
optimizer = torch.optim.AdamW(policy_model.parameters(), lr=1e-6)
for batch in dataloader: # batch contains chosen/rejected pairs
# Forward pass through both models
policy_chosen_logps = compute_log_probs(policy_model, batch['chosen'])
policy_rejected_logps = compute_log_probs(policy_model, batch['rejected'])
with torch.no_grad():
ref_chosen_logps = compute_log_probs(reference_model, batch['chosen'])
ref_rejected_logps = compute_log_probs(reference_model, batch['rejected'])
# Compute loss and update
loss = dpo_loss(policy_chosen_logps, policy_rejected_logps,
ref_chosen_logps, ref_rejected_logps)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Key Takeaway: DPO training requires maintaining two models—a trainable policy and frozen reference—computing preference-based loss on paired completions, and applying standard gradient descent to optimize only the policy model.