Course contentsShow
← Machine Learning and Deep Learning
Lesson 2289 of 3,53850. Deep Reinforcement Learning: Advanced Policy MethodsFree lesson

Limitations of Basic Policy Gradient Methods

Understanding instability, large update steps, and catastrophic forgetting that motivate trust region approaches.

Limitations of Basic Policy Gradient Methods

What you'll learn: Why vanilla policy gradient methods struggle with training instability and what problems motivated more advanced approaches.

The Core Problem

Basic policy gradient methods like REINFORCE and simple actor-critic algorithms suffer from three critical weaknesses that make them unreliable for complex tasks:

1. Training Instability

Policy gradients update parameters directly based on sampled trajectories. A few bad samples can push your policy in wildly different directions between updates. Unlike supervised learning where your training data is fixed, each policy change alters the data distribution you'll see next—creating a moving target that's hard to track.

Analogy: Imagine learning to ride a bike, but every time you adjust your balance, the bike's geometry randomly changes. You might overcorrect and fall off entirely.

2. Large, Destructive Update Steps

Standard gradient descent with a fixed learning rate can take steps that are too large for the policy space. A single bad update might collapse a previously working policy into one that performs catastrophically worse—and there's no easy way to recover.

3. Catastrophic Forgetting

When your policy dramatically changes, it "forgets" what worked before. The agent might have learned to navigate one part of the environment well, but a large update erases that knowledge while trying to improve elsewhere. This leads to oscillating performance rather than steady improvement.

Why This Matters

These limitations mean basic methods require extremely careful hyperparameter tuning (tiny learning rates, just right batch sizes) and still frequently fail to converge on challenging problems. This instability is precisely what motivated trust region methods that constrain how much the policy can change per update.

Key Takeaway: Basic policy gradient methods are fragile because they lack mechanisms to prevent destructive updates, leading to instability, catastrophic forgetting, and unreliable training—problems that trust region approaches directly address.