Supervised Fine-Tuning Mechanics
What you'll learn: How the supervised fine-tuning process trains models to generate assistant-like responses from instruction prompts.
The Core Training Loop
Supervised Fine-Tuning (SFT) is where a base language model learns to follow instructions by predicting assistant responses given instruction-response pairs. Unlike standard language modeling that predicts all tokens equally, SFT focuses the loss only on the assistant's response tokens.
Think of it like teaching someone to answer questions: you show them the question (instruction) and the correct answer (response), but you only grade them on whether they got the answer right—not on whether they repeated the question correctly.
How the Mechanics Work
During SFT, each training example consists of:
- Input tokens: The instruction/prompt (sometimes with system message)
- Target tokens: The assistant's response
The key mechanic: you feed the entire sequence (instruction + response) through the model, but you mask the loss for instruction tokens. The model only receives gradient updates for predicting the response portion correctly.
For example:
- Full sequence:
"Instruction: Summarize this text. Response: Here is a summary..." - Loss computed: Only on
"Here is a summary..."tokens - Gradients backpropagate through all layers (as in full fine-tuning or PEFT)
The forward pass processes everything, but the loss function ignores instruction positions by setting their loss contribution to zero (or using attention masks). This teaches the model: "when you see an instruction format, generate appropriate assistant responses."
Training Implementation
The optimizer (like AdamW) updates weights based only on response-token losses, using the learning rates and training duration strategies you've already learned. Whether using full fine-tuning, LoRA, or QLoRA, the SFT mechanics remain the same—only the response tokens matter for optimization.
Key Takeaway: Supervised fine-tuning teaches instruction-following by computing loss exclusively on assistant response tokens while showing the model complete instruction-response examples during the forward pass.