Course contentsShow
← Machine Learning and Deep Learning
Lesson 1816 of 3,53838. Instruction Tuning and AlignmentFree lesson

Iterative DPO and Online Alignment

Generating new preference data from current policy to continuously improve alignment.

Iterative DPO and Online Alignment

You'll learn how to continuously improve model alignment by generating fresh preference data from your evolving policy, creating a feedback loop that goes beyond single-round training.

The Core Idea

Standard DPO trains once on a fixed preference dataset—but what if your model could learn from its own outputs? Iterative DPO means running multiple rounds where you:

  1. Generate new responses from your current policy
  2. Collect human (or AI) preferences on these fresh outputs
  3. Train another round of DPO using this new data
  4. Repeat

This is online alignment because the model adapts to data it generates during training, rather than only learning from static, pre-collected examples.

Why It Matters

Your initial preference dataset reflects what an older model version produced. As your policy improves, it explores different parts of the output space. By generating new samples and getting feedback on those, you:

  • Cover blind spots the original dataset missed
  • Prevent distribution shift between training data and actual model behavior
  • Discover new failure modes that emerge only after initial alignment

Think of it like teaching someone to cook: you don't just critique their first attempt forever—you taste each improved dish and give updated feedback.

The Iteration Loop

Each cycle refines alignment:

  • Round 1: Train DPO on initial preference pairs (from SFT model outputs)
  • Round 2: Generate responses with DPO-v1 model → collect new preferences → train DPO-v2
  • Round 3+: Repeat, using the latest policy as the data generator

The reference model typically updates each round (set to the previous iteration's policy) to maintain meaningful KL penalties.

Key Takeaway: Iterative DPO creates a virtuous cycle where the model learns from its own improving outputs, enabling continuous alignment refinement beyond what static datasets allow.