Course contentsShow
← Machine Learning and Deep Learning
Lesson 2308 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Multiple Epochs of Updates

Why PPO performs multiple gradient steps on the same batch of experience, unlike on-policy methods.

Multiple Epochs of Updates

What you'll learn: Why PPO breaks from traditional on-policy rules by reusing the same batch of experience multiple times.

The On-Policy Dilemma

Traditional on-policy methods like vanilla policy gradients have a strict rule: collect experience with your current policy, update once, then throw that data away. Why? Because after you update your policy, that experience was generated by an old policy—it's now "stale" and theoretically invalid for further learning.

This is wasteful. Collecting experience (especially in simulation or real environments) is expensive. Using each sample only once feels like reading a textbook page and immediately burning it.

PPO's Clever Workaround

PPO asks: "What if we could squeeze more learning from each batch without breaking things?"

The clipped surrogate objective makes this possible. Remember the probability ratio π_new(a|s) / π_old(a|s)? The clipping mechanism constrains how far the new policy can drift from the old one—typically within [1-ε, 1+ε], where ε ≈ 0.2.

This means PPO can perform multiple epochs (often 3-10) of gradient updates on the same batch:

  1. Collect batch with current policy
  2. For epoch 1, 2, 3, ..., K:
    • Compute gradients using clipped objective
    • Update policy and value function
  3. Collect new batch, repeat

The clipping acts as training wheels, preventing destructive updates even when the policy has changed somewhat since data collection.

Why This Matters

Multiple epochs dramatically improve sample efficiency—you extract more learning signal per environment interaction. Instead of one gradient step per experience, you might take 10, effectively multiplying your data's value.

The tradeoff? You're slightly off-policy after the first update. But empirically, the clipping keeps you safe, and the gains massively outweigh this theoretical impurity.

Key Takeaway: PPO performs multiple gradient updates per batch of experience by using the clipped objective as a safety mechanism, dramatically improving sample efficiency compared to strict on-policy methods that discard data after one use.