Mini-Batch Updates and Multiple Epochs
What you'll learn: How PPO efficiently reuses collected rollout data through multiple gradient updates while maintaining training stability.
The Core Challenge
In RLHF, generating rollouts from your LLM is expensive—you must run forward passes to produce responses, compute rewards, and calculate advantages. Simply throwing away this data after one gradient update would be wasteful. But naive reuse can violate the trust region constraint and destabilize training.
How PPO Solves This
PPO's clipped surrogate objective enables multiple passes over the same batch of rollouts. Instead of the traditional RL approach (one update per sample, then discard), PPO:
- Collects a batch of rollouts (e.g., 256 prompts with responses, rewards, and advantages)
- Splits into mini-batches (e.g., 32 samples each)
- Performs several epochs (e.g., 4 passes) of gradient updates on these mini-batches
Why This Works
The clipped objective naturally limits how much the policy can change from the reference policy that generated the rollouts. Even when reusing data, the clipping prevents destructive updates that would invalidate the advantage estimates.
Think of it like learning from a practice exam: you can review the same questions multiple times, but each review yields diminishing returns. PPO clips away updates that would "over-learn" from stale data.
Practical Settings
- Mini-batch size: 32-64 samples
- Epochs: 3-5 (more risks overfitting to old policy data)
- Total updates per rollout: ~4-8× more efficient than single-update RL
Key Takeaway: PPO's mini-batch updates and multiple epochs dramatically improve sample efficiency by safely reusing expensive rollouts, with the clipped objective ensuring each update respects trust region constraints even when training on slightly outdated data.