Course contentsShow
← Machine Learning and Deep Learning
Lesson 2255 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

Variance in Policy Gradients

Why policy gradient estimates have high variance and how this impacts learning stability.

Variance in Policy Gradients

What You'll Learn

You'll understand why policy gradient estimates are notoriously noisy and how this instability can derail your reinforcement learning agent's training.

The Core Problem

When you use policy gradients, you're estimating the gradient by sampling episodes from your stochastic policy. The problem? Different episode rollouts produce wildly different gradient estimates, even when nothing about your policy has changed.

Think of it like estimating the average height of people in a city by randomly sampling small groups. Some samples might catch a basketball team (very tall average), others might catch a kindergarten class (very short). Your estimates swing dramatically—that's high variance.

Why Does This Happen?

In episode-based gradient estimation, your gradient update depends on:

  • The trajectory you happened to sample: Random actions lead to different states
  • The cumulative rewards: Two similar trajectories can yield drastically different total returns due to environment stochasticity
  • The length of episodes: Longer episodes compound uncertainty

The score function estimator multiplies rewards by log-probabilities of actions. When rewards vary significantly across episodes (even with the same policy), your gradient estimates become unreliable.

Impact on Learning

High variance means:

  • Slow convergence: Your policy parameters jump around instead of steadily improving
  • Wasted samples: Many rollouts contribute misleading gradient signals
  • Training instability: The agent might unlearn good behaviors due to noisy updates

It's like trying to walk toward a destination while someone randomly pushes you left or right—you eventually get there, but the path is chaotic and inefficient.

Key Takeaway: Policy gradient estimates suffer from high variance because they rely on noisy episode samples, causing unstable learning that requires large amounts of data or variance-reduction techniques to overcome.