DPO Hyperparameters: Beta and Learning Rate
What you'll learn: How two critical hyperparameters—beta and learning rate—control DPO training stability and alignment quality.
Understanding Beta (β)
Beta is the KL divergence penalty coefficient that controls how much your fine-tuned policy model can deviate from the reference model during DPO training.
Think of beta as a "leash length":
- Higher beta (e.g., 0.5): Tight leash. The policy stays close to the reference model, leading to conservative, safer responses but potentially slower learning
- Lower beta (e.g., 0.01): Loose leash. The policy can explore more freely, learning preferences faster but risking reward hacking or unstable behavior
Typical values: Beta usually ranges from 0.1 to 0.5, with 0.1 being a common starting point. The optimal value depends on your preference data quality and how much deviation you can tolerate.
Learning Rate Considerations
DPO learning rates are typically lower than supervised fine-tuning because you're optimizing against a reference model rather than just fitting data.
Common schedule:
- Start:
1e-6to5e-6(much smaller than SFT's1e-5) - Use linear decay or cosine schedule to gradually reduce learning rate
- Short warmup (few hundred steps) helps stabilize early training
Why so conservative? DPO directly optimizes preference rankings, and aggressive updates can cause the policy to diverge too quickly from the reference, breaking the implicit reward model that DPO relies on.
Balancing Both
These hyperparameters work together: if you increase beta (tighter constraint), you might afford a slightly higher learning rate. If you lower beta, reduce learning rate proportionally to maintain stability.
Key Takeaway: Beta controls deviation from the reference model (typically 0.1–0.5), while DPO learning rates should be conservative (1e-6 to 5e-6) to prevent destabilizing the implicit reward function underlying preference optimization.