The Three-Stage RLHF Pipeline
What you'll learn: How RLHF breaks down into three distinct stages—supervised fine-tuning, reward model training, and reinforcement learning optimization—to align language models with human preferences.
The Big Picture
RLHF isn't a single monolithic process. It's a carefully orchestrated pipeline of three stages, each building on the previous one. Think of it like preparing a chef: first they learn recipes (supervised fine-tuning), then they learn what makes food taste good to diners (reward modeling), and finally they practice optimizing dishes based on that taste feedback (RL optimization).
Stage 1: Supervised Fine-Tuning (SFT)
You start with your base model and perform instruction tuning using high-quality demonstration data. Human annotators write ideal responses to prompts, and you train the model to imitate these examples. This creates your SFT model—a competent starting point that can follow instructions reasonably well.
Stage 2: Reward Model Training
Next, you train a separate reward model to predict human preferences. Annotators rank multiple responses to the same prompt (e.g., "Response A is better than Response B"). You train a model—often initialized from your SFT model—to output scalar scores that match these preference rankings. This reward model becomes your automated "judge" of response quality.
Stage 3: RL Optimization
Finally, you use reinforcement learning (typically PPO—Proximal Policy Optimization) to optimize your SFT model using the reward model as feedback. The model generates responses, the reward model scores them, and RL updates the policy to maximize reward while staying close to the SFT model (to prevent degradation).
Key Takeaway: RLHF's three-stage pipeline—SFT for basic competence, reward modeling to capture human preferences, and RL optimization to maximize those preferences—transforms base models into aligned assistants that better match human values and expectations.