Course contentsShow
← Machine Learning and Deep Learning
Lesson 1803 of 3,53838. Instruction Tuning and AlignmentFree lesson

Limitations of RLHF and the Need for DPO

Why RLHF is complex, unstable, and computationally expensive, motivating simpler alignment methods.

Limitations of RLHF and the Need for DPO

What You'll Learn

You'll understand why RLHF, despite its success, suffers from complexity, instability, and computational expense—motivating the development of simpler alignment methods like DPO.

The Problem with RLHF's Three-Stage Pipeline

RLHF works, but it's complicated. Recall the three stages: supervised fine-tuning, reward model training, and PPO-based RL fine-tuning. Each stage introduces its own challenges:

1. Multi-Model Training Overhead
You need to train and maintain multiple models simultaneously: the policy model being optimized, a frozen reference model for KL divergence, a reward model for scoring, and often a value function network. This quadruples your memory requirements.

2. PPO's Inherent Instability
PPO hyperparameters (clip ratio, KL coefficient, learning rate) are notoriously sensitive. Too aggressive, and you get reward hacking. Too conservative, and learning stalls. Finding the right balance requires extensive tuning and compute.

3. Sampling and Rollout Costs
Every training step requires generating full text sequences from your policy to compute rewards. This is slow—imagine running inference thousands of times just to take one gradient step.

The Analogy

Think of RLHF like learning to cook by: (1) taking a cooking class, (2) hiring a food critic to rate your dishes, then (3) cooking thousands of trial meals, getting scored after each one, and adjusting your technique incrementally. It works, but wouldn't it be simpler if you could just learn directly from comparing recipes people prefer?

Why We Need Simpler Methods

The complexity and instability motivated researchers to ask: Can we skip the RL loop entirely and optimize preferences directly? This question led to Direct Preference Optimization (DPO) and similar approaches that eliminate the reward model and PPO machinery while achieving comparable alignment quality.

Key Takeaway: RLHF's multi-stage pipeline, computational expense, and training instability created demand for simpler, more stable alignment methods that optimize preferences directly without explicit reinforcement learning.