RLHF vs Supervised Fine-Tuning Trade-offs
You'll learn when to choose RLHF over SFT alone and understand the computational costs of each approach.
The Core Decision
After instruction tuning with supervised fine-tuning (SFT), you face a choice: stop there or invest in RLHF. Both approaches align models to follow instructions, but they differ fundamentally in how they learn and what they optimize.
SFT trains on direct examples—"here's the input, here's the correct output." It's straightforward: compute loss on demonstration data, backpropagate, update weights. You need quality instruction-response pairs, but the training loop is familiar and efficient.
RLHF goes further by learning from preferences rather than demonstrations. It trains a reward model, then uses RL to optimize for human-preferred outputs. This captures nuanced preferences that are hard to demonstrate explicitly—like helpfulness, safety, or tone—but at a significant cost.
When RLHF Provides Benefits
RLHF excels when:
- Preferences matter more than correctness: Creative writing, conversational style, safety boundaries
- Demonstrations are insufficient: It's easier to rank outputs than write perfect examples
- Avoiding reward hacking: You want the model to optimize what humans actually want, not just pattern-match training data
- Fine-grained quality control: Steering behavior beyond what SFT examples can capture
Computational Cost Considerations
SFT costs: Single model training pass, standard supervised learning, moderate memory requirements.
RLHF costs: Train reward model first, then maintain two copies of the large model (policy and reference), compute KL divergence penalties, sample multiple outputs per prompt during RL training. Roughly 3-5x more expensive than SFT alone in compute and memory.
Key Takeaway: Use SFT when you have good demonstrations and clear right answers; add RLHF when capturing human preferences and nuanced quality matters enough to justify 3-5x higher computational costs.