Course contentsShow
← Machine Learning and Deep Learning
Lesson 1802 of 3,53838. Instruction Tuning and AlignmentFree lesson

PPO Alternatives and Recent Improvements

Exploring variants like GRPO, rejection sampling, and online DPO that simplify the RLHF pipeline.

PPO Alternatives and Recent Improvements

You'll learn about simplified alternatives to PPO that make RLHF training more efficient and stable.

The Problem with PPO

By now you've seen PPO's complexity: value functions, advantage estimation, clipped objectives, rollouts, and careful hyperparameter tuning. While PPO works, researchers have asked: Do we really need all this machinery?

Simpler Approaches Emerging

Recent methods simplify or even eliminate the RL loop entirely:

GRPO (Group Relative Policy Optimization)

Instead of training a separate value function, GRPO samples multiple responses per prompt and uses their relative rewards to estimate advantages. Think of it like grading on a curve within each batch—you don't need absolute value estimates, just relative rankings.

Rejection Sampling Fine-Tuning

Generate many responses, filter using your reward model, then simply supervised-fine-tune on the best ones. No RL required! It's like curating a high-quality dataset using your reward model as a filter, then doing traditional SFT.

Online DPO (Direct Preference Optimization)

Traditional DPO (which you'll learn soon) works offline on fixed preference data. Online DPO generates fresh preference pairs during training, using the current policy to sample responses and comparing them. This combines DPO's simplicity with online learning's adaptability—without needing a separate reward model.

Why These Matter

These alternatives address PPO's pain points: simpler implementation, fewer hyperparameters, no value network to train, and often better stability. They maintain the core idea—optimize policy based on quality signals—while cutting computational overhead.

Key Takeaway: Modern RLHF explores simpler alternatives to PPO like GRPO, rejection sampling, and online DPO that achieve comparable alignment with less complexity and fewer moving parts.