Hyperparameters: Clip Ratio and KL Coefficient
What You'll Learn
You'll learn how to tune the two most critical hyperparameters in PPO-based RLHF: the clip ratio (ε) and the KL coefficient (β), which together control how aggressively your language model updates during alignment training.
The Balancing Act
When training LLMs with PPO, you're walking a tightrope. Update too aggressively and your model forgets what it learned or exploits the reward model (reward hacking). Update too conservatively and training crawls at a snail's pace. Two hyperparameters give you precise control over this balance:
The Clip Ratio (ε)
The clip ratio (typically 0.1 to 0.3) defines the maximum policy change PPO allows per update step. Think of it as a speed limiter on your car—it prevents you from accelerating so fast that you lose control. A clip ratio of 0.2 means the new policy can only be 20% more or less likely to generate a token compared to the old policy before PPO "clips" the gradient and stops learning from that example.
- Lower values (0.1): More stable, slower learning, safer for production
- Higher values (0.3): Faster learning, riskier, more prone to instability
The KL Coefficient (β)
The KL coefficient (typically 0.01 to 0.1) controls how heavily you penalize divergence from the reference model. It's your safety tether—keeping the updated policy from drifting too far from the original SFT model. A higher β means you value staying close to the reference model, while a lower β gives the policy more freedom to explore.
- Lower values (0.01): More aggressive updates, faster alignment, higher drift risk
- Higher values (0.1): Conservative updates, maintains base capabilities better
Tuning in Practice
Start with standard values (ε=0.2, β=0.02) and monitor two signals: reward model scores (should increase) and KL divergence (should stay bounded, typically < 5-10). If KL explodes, increase β or decrease ε. If learning stalls, try the opposite.
Key Takeaway: The clip ratio and KL coefficient are your primary levers for balancing learning speed against training stability—tune them based on your reward progression and KL divergence metrics.