Course contentsShow
← Machine Learning and Deep Learning
Lesson 1766 of 3,53838. Instruction Tuning and AlignmentFree lesson

The Role of the SFT Model in RLHF

Why supervised fine-tuning on high-quality demonstrations precedes reward modeling and provides initialization.

The Role of the SFT Model in RLHF

What You'll Learn

You'll understand why RLHF begins with a supervised fine-tuned (SFT) model rather than training directly from the base model.

Why Start with SFT?

In the three-stage RLHF pipeline, supervised fine-tuning comes first for a critical reason: it provides a high-quality initialization point before the complexity of reinforcement learning begins.

Think of it like teaching a student to write essays. You wouldn't start by just giving vague feedback like "this essay is better than that one." First, you'd show them examples of good essays to establish basic competency. That's exactly what SFT does for language models.

The Foundation Layer

The SFT model serves as the reference point for the entire RLHF process:

  1. Quality Baseline: By training on high-quality human demonstrations (instruction-response pairs), the model learns what good outputs look like before learning what outputs humans prefer.

  2. Behavioral Initialization: The SFT model already follows instructions reasonably well, making it easier for the reward model to distinguish subtle preference differences rather than basic competence.

  3. Training Stability: Starting from a random or base model would make reward modeling and RL training extremely unstable—the model might generate nonsense that's hard to rank consistently.

  4. Efficiency: The SFT model narrows the output space to reasonable responses, so RL can focus on refining behavior rather than discovering basic language patterns from scratch.

The Handoff

The SFT model essentially says: "Here's how to respond helpfully." Then reward modeling and RL say: "Now let's optimize for what humans actually prefer among helpful responses."

Key Takeaway: The SFT model in RLHF provides a competent, instruction-following initialization that makes subsequent reward modeling and reinforcement learning tractable, stable, and focused on preference optimization rather than basic capability learning.