Course contentsShow
← Machine Learning and Deep Learning
Lesson 1813 of 3,53838. Instruction Tuning and AlignmentFree lesson

Implementing DPO in PyTorch

Code walkthrough: loading model pairs, computing loss, and training with gradient descent.

Implementing DPO in PyTorch

You'll learn how to build a complete DPO training loop in PyTorch, from loading model pairs to computing the preference loss and updating weights.

Core Idea

Direct Preference Optimization requires three components: a policy model (being trained), a reference model (frozen copy), and paired preference data. Your training loop computes log probabilities for chosen and rejected completions under both models, calculates the DPO loss using the beta hyperparameter, and updates only the policy model.

Code Walkthrough

Loading Model Pairs

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load policy model (trainable)
policy_model = AutoModelForCausalLM.from_pretrained("model_name")
policy_model.train()

# Load reference model (frozen)
reference_model = AutoModelForCausalLM.from_pretrained("model_name")
reference_model.eval()
for param in reference_model.parameters():
    param.requires_grad = False

Computing DPO Loss

import torch
import torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             reference_chosen_logps, reference_rejected_logps, beta=0.1):
    """Compute DPO loss from log probabilities"""
    policy_logratios = policy_chosen_logps - policy_rejected_logps
    reference_logratios = reference_chosen_logps - reference_rejected_logps
    
    losses = -F.logsigmoid(beta * (policy_logratios - reference_logratios))
    return losses.mean()

Training Loop

optimizer = torch.optim.AdamW(policy_model.parameters(), lr=1e-6)

for batch in dataloader:  # batch contains chosen/rejected pairs
    # Forward pass through both models
    policy_chosen_logps = compute_log_probs(policy_model, batch['chosen'])
    policy_rejected_logps = compute_log_probs(policy_model, batch['rejected'])
    
    with torch.no_grad():
        ref_chosen_logps = compute_log_probs(reference_model, batch['chosen'])
        ref_rejected_logps = compute_log_probs(reference_model, batch['rejected'])
    
    # Compute loss and update
    loss = dpo_loss(policy_chosen_logps, policy_rejected_logps,
                    ref_chosen_logps, ref_rejected_logps)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Key Takeaway: DPO training requires maintaining two models—a trainable policy and frozen reference—computing preference-based loss on paired completions, and applying standard gradient descent to optimize only the policy model.