The Policy Update Problem
What you'll learn: Why taking large, unrestricted steps when updating your policy can catastrophically destroy performance in reinforcement learning.
The Core Problem
When training policy gradient methods, we compute gradients that point toward better performance and update our policy parameters accordingly. But here's the catch: the gradient is only valid locally, near the current policy.
If you take too large a step (using a high learning rate), you can jump to a drastically different policy where:
- Your performance estimates are no longer accurate
- The new policy might perform far worse than before
- Recovery becomes difficult because you've "forgotten" what worked
This is worse than supervised learning because in RL, the data distribution depends on your policy. Change your policy too much, and the data you collected becomes irrelevant—potentially causing a performance collapse where your agent suddenly fails at tasks it previously mastered.
Why This Happens
Think of it like hiking in fog with a compass. The compass (gradient) tells you which direction is uphill right where you stand. If you take a small step and check again, you're probably still going uphill. But if you sprint blindly in that direction, you might run off a cliff—the terrain changed, but you assumed it stayed the same.
In mathematical terms: policy gradients assume a linear approximation of the objective function. This approximation breaks down when the policy changes significantly.
The Solution: Constrained Optimization
Instead of unrestricted gradient descent, we need constrained policy updates that limit how much the policy can change in one step. This leads to trust region methods and more stable training.
Key Takeaway: Unrestricted policy updates can cause catastrophic performance collapse because gradients are only locally valid; we need constrained optimization to ensure safe, monotonic improvement.