Course contentsShow
← Machine Learning and Deep Learning
Lesson 1761 of 3,53838. Instruction Tuning and AlignmentFree lesson

What is Reinforcement Learning from Human Feedback (RLHF)?

Overview of RLHF as a technique to align language models with human preferences through reward modeling and RL.

What is Reinforcement Learning from Human Feedback (RLHF)?

You'll learn: How RLHF aligns language models with human preferences by combining reward models and reinforcement learning.

The Core Challenge

Instruction tuning teaches models to follow instructions, but it doesn't guarantee outputs that humans actually prefer. A model might generate factually correct but unhelpful responses, or technically follow instructions while being rude or unsafe. RLHF solves this by optimizing for human preferences directly.

How RLHF Works

RLHF extends instruction tuning with three key stages:

1. Supervised Fine-Tuning (SFT)

Start with an instruction-tuned model—you've already learned this step. This provides the baseline capability.

2. Reward Model Training

Collect human preference data: given the same prompt, humans rank multiple model outputs (e.g., "Output A is better than Output B"). Train a reward model—typically another language model—to predict which outputs humans prefer. This model learns to score responses on a numeric scale reflecting human judgment.

3. Reinforcement Learning Optimization

Use the reward model as an automated judge. The language model generates responses, the reward model scores them, and reinforcement learning algorithms (typically PPO—Proximal Policy Optimization) adjust the model's parameters to maximize reward scores. This is like training a model to "play a game" where the goal is generating high-scoring, human-preferred outputs.

The Analogy

Think of RLHF like training a chef: instruction tuning teaches recipes (mechanics), but RLHF teaches taste—what makes food actually delicious to diners, not just technically correct.

Key Takeaway: RLHF aligns language models with nuanced human preferences by training a reward model from human comparisons, then using reinforcement learning to optimize the language model against that learned reward signal—going beyond instruction-following to preference-matching.