Limitations of Basic Policy Gradient Methods
What you'll learn: Why vanilla policy gradient methods struggle with training instability and what problems motivated more advanced approaches.
The Core Problem
Basic policy gradient methods like REINFORCE and simple actor-critic algorithms suffer from three critical weaknesses that make them unreliable for complex tasks:
1. Training Instability
Policy gradients update parameters directly based on sampled trajectories. A few bad samples can push your policy in wildly different directions between updates. Unlike supervised learning where your training data is fixed, each policy change alters the data distribution you'll see next—creating a moving target that's hard to track.
Analogy: Imagine learning to ride a bike, but every time you adjust your balance, the bike's geometry randomly changes. You might overcorrect and fall off entirely.
2. Large, Destructive Update Steps
Standard gradient descent with a fixed learning rate can take steps that are too large for the policy space. A single bad update might collapse a previously working policy into one that performs catastrophically worse—and there's no easy way to recover.
3. Catastrophic Forgetting
When your policy dramatically changes, it "forgets" what worked before. The agent might have learned to navigate one part of the environment well, but a large update erases that knowledge while trying to improve elsewhere. This leads to oscillating performance rather than steady improvement.
Why This Matters
These limitations mean basic methods require extremely careful hyperparameter tuning (tiny learning rates, just right batch sizes) and still frequently fail to converge on challenging problems. This instability is precisely what motivated trust region methods that constrain how much the policy can change per update.
Key Takeaway: Basic policy gradient methods are fragile because they lack mechanisms to prevent destructive updates, leading to instability, catastrophic forgetting, and unreliable training—problems that trust region approaches directly address.