Course contentsShow
← Machine Learning and Deep Learning
Lesson 1798 of 3,53838. Instruction Tuning and AlignmentFree lesson

Hyperparameters: Clip Ratio and KL Coefficient

Tuning the clipping threshold and KL penalty weight to balance learning speed and stability.

Hyperparameters: Clip Ratio and KL Coefficient

What You'll Learn

You'll learn how to tune the two most critical hyperparameters in PPO-based RLHF: the clip ratio (ε) and the KL coefficient (β), which together control how aggressively your language model updates during alignment training.

The Balancing Act

When training LLMs with PPO, you're walking a tightrope. Update too aggressively and your model forgets what it learned or exploits the reward model (reward hacking). Update too conservatively and training crawls at a snail's pace. Two hyperparameters give you precise control over this balance:

The Clip Ratio (ε)

The clip ratio (typically 0.1 to 0.3) defines the maximum policy change PPO allows per update step. Think of it as a speed limiter on your car—it prevents you from accelerating so fast that you lose control. A clip ratio of 0.2 means the new policy can only be 20% more or less likely to generate a token compared to the old policy before PPO "clips" the gradient and stops learning from that example.

  • Lower values (0.1): More stable, slower learning, safer for production
  • Higher values (0.3): Faster learning, riskier, more prone to instability

The KL Coefficient (β)

The KL coefficient (typically 0.01 to 0.1) controls how heavily you penalize divergence from the reference model. It's your safety tether—keeping the updated policy from drifting too far from the original SFT model. A higher β means you value staying close to the reference model, while a lower β gives the policy more freedom to explore.

  • Lower values (0.01): More aggressive updates, faster alignment, higher drift risk
  • Higher values (0.1): Conservative updates, maintains base capabilities better

Tuning in Practice

Start with standard values (ε=0.2, β=0.02) and monitor two signals: reward model scores (should increase) and KL divergence (should stay bounded, typically < 5-10). If KL explodes, increase β or decrease ε. If learning stalls, try the opposite.

Key Takeaway: The clip ratio and KL coefficient are your primary levers for balancing learning speed against training stability—tune them based on your reward progression and KL divergence metrics.