The Role of the SFT Model in RLHF
What You'll Learn
You'll understand why RLHF begins with a supervised fine-tuned (SFT) model rather than training directly from the base model.
Why Start with SFT?
In the three-stage RLHF pipeline, supervised fine-tuning comes first for a critical reason: it provides a high-quality initialization point before the complexity of reinforcement learning begins.
Think of it like teaching a student to write essays. You wouldn't start by just giving vague feedback like "this essay is better than that one." First, you'd show them examples of good essays to establish basic competency. That's exactly what SFT does for language models.
The Foundation Layer
The SFT model serves as the reference point for the entire RLHF process:
-
Quality Baseline: By training on high-quality human demonstrations (instruction-response pairs), the model learns what good outputs look like before learning what outputs humans prefer.
-
Behavioral Initialization: The SFT model already follows instructions reasonably well, making it easier for the reward model to distinguish subtle preference differences rather than basic competence.
-
Training Stability: Starting from a random or base model would make reward modeling and RL training extremely unstable—the model might generate nonsense that's hard to rank consistently.
-
Efficiency: The SFT model narrows the output space to reasonable responses, so RL can focus on refining behavior rather than discovering basic language patterns from scratch.
The Handoff
The SFT model essentially says: "Here's how to respond helpfully." Then reward modeling and RL say: "Now let's optimize for what humans actually prefer among helpful responses."
Key Takeaway: The SFT model in RLHF provides a competent, instruction-following initialization that makes subsequent reward modeling and reinforcement learning tractable, stable, and focused on preference optimization rather than basic capability learning.