Course contentsShow
← Machine Learning and Deep Learning
Lesson 1775 of 3,53838. Instruction Tuning and AlignmentFree lesson

Iterative RLHF and Online Learning

Continuously updating reward models with new data from the evolving policy to improve alignment over time.

Iterative RLHF and Online Learning

What you'll learn: How to continuously improve alignment by updating reward models with fresh data from your evolving policy.

The Core Idea

In basic RLHF, you train a reward model once on static human preferences, then use it to fine-tune your policy. But there's a problem: as your policy improves and generates different kinds of responses, the reward model becomes outdated—it was trained on outputs from an older version of your model.

Iterative RLHF solves this by treating alignment as an ongoing cycle rather than a one-time process. After each round of RL fine-tuning, you:

  1. Generate new outputs with your improved policy
  2. Collect fresh human preferences on these outputs
  3. Retrain or update your reward model with this new data
  4. Use the updated reward model for another round of RL fine-tuning

This creates a feedback loop where the reward model "chases" the policy's evolution, staying relevant to the distribution of responses your model actually produces.

Why This Matters

Think of it like teaching someone to cook: you don't give them all feedback upfront based on dishes they haven't made yet. You taste what they cook now, give feedback, they improve, then you taste their new creations and adjust your guidance accordingly.

Without iterative updates, your reward model may incorrectly score novel outputs from your improved policy, leading to reward hacking or misalignment. Online learning keeps the reward signal grounded in your model's current capabilities and tendencies.

The Challenge

This approach requires continuous human annotation as your model evolves, making it resource-intensive. You're also constantly retraining components, which adds computational cost.

Key Takeaway: Iterative RLHF continuously refreshes your reward model with new policy outputs, preventing staleness and maintaining alignment as your model evolves—but at the cost of ongoing human feedback collection.