Course contentsShow
← AI Engineering
Lesson 850 of 1,88621. Human Evaluation and FeedbackFree lesson

The Three Stages of RLHF

Overview of supervised fine-tuning, reward model training, and reinforcement learning optimization in the RLHF pipeline.

The Three Stages of RLHF

What you'll learn: You'll understand the three distinct phases that transform a base language model into one that aligns with human preferences through RLHF.

The Pipeline Overview

RLHF isn't a single technique—it's a three-stage pipeline where each phase builds on the previous one. Think of it like training an athlete: first you teach the fundamentals, then you show them what "good" looks like, and finally you let them practice with feedback.

Stage 1: Supervised Fine-Tuning (SFT)

You start by collecting high-quality demonstrations—examples of ideal model behavior written by humans. The base model is fine-tuned on these examples using standard supervised learning. This gives the model a foundation of how to respond helpfully, like teaching a student with textbook examples.

Stage 2: Reward Model Training

Next, you train a separate reward model that learns to score outputs. Annotators compare multiple model responses and indicate which they prefer. The reward model learns to predict human preferences, becoming an automated judge that can evaluate millions of outputs—far more than humans could review.

Stage 3: Reinforcement Learning Optimization

Finally, you use the reward model to optimize the original model through reinforcement learning (typically PPO - Proximal Policy Optimization). The model generates responses, the reward model scores them, and the model adjusts to maximize those scores. It's learning by trial and error, guided by the reward signal you've taught it to optimize for.

Each stage depends on the previous: you need good initial behavior (SFT) before you can collect meaningful comparisons, and you need a reliable reward model before you can optimize with RL.

Key Takeaway: RLHF consists of three sequential stages—supervised fine-tuning to establish baseline behavior, reward model training to capture human preferences, and reinforcement learning to optimize the model against those learned preferences.