Variance in Policy Gradients
What You'll Learn
You'll understand why policy gradient estimates are notoriously noisy and how this instability can derail your reinforcement learning agent's training.
The Core Problem
When you use policy gradients, you're estimating the gradient by sampling episodes from your stochastic policy. The problem? Different episode rollouts produce wildly different gradient estimates, even when nothing about your policy has changed.
Think of it like estimating the average height of people in a city by randomly sampling small groups. Some samples might catch a basketball team (very tall average), others might catch a kindergarten class (very short). Your estimates swing dramatically—that's high variance.
Why Does This Happen?
In episode-based gradient estimation, your gradient update depends on:
- The trajectory you happened to sample: Random actions lead to different states
- The cumulative rewards: Two similar trajectories can yield drastically different total returns due to environment stochasticity
- The length of episodes: Longer episodes compound uncertainty
The score function estimator multiplies rewards by log-probabilities of actions. When rewards vary significantly across episodes (even with the same policy), your gradient estimates become unreliable.
Impact on Learning
High variance means:
- Slow convergence: Your policy parameters jump around instead of steadily improving
- Wasted samples: Many rollouts contribute misleading gradient signals
- Training instability: The agent might unlearn good behaviors due to noisy updates
It's like trying to walk toward a destination while someone randomly pushes you left or right—you eventually get there, but the path is chaotic and inefficient.
Key Takeaway: Policy gradient estimates suffer from high variance because they rely on noisy episode samples, causing unstable learning that requires large amounts of data or variance-reduction techniques to overcome.