Course contentsShow
← Machine Learning and Deep Learning
Lesson 2273 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

High Variance Problem in REINFORCE

Understanding why pure Monte Carlo estimates create noisy gradients and slow learning in practice.

High Variance Problem in REINFORCE

What You'll Learn

You'll understand why REINFORCE's pure Monte Carlo approach creates extremely noisy gradient estimates that dramatically slow down learning.

The Core Problem

REINFORCE uses full episode returns to estimate policy gradients. This means it waits until an entire episode completes, then calculates the total return (sum of all future rewards) from each timestep. While this gives an unbiased estimate, it introduces massive variance into the gradient updates.

Think of it like trying to steer a ship by only checking your position after a complete journey. If you arrive off-course, was it the first turn that was wrong? The middle part? The end? Every small random fluctuation in the environment affects your final return, making it hard to identify which actions were truly good or bad.

Why Monte Carlo Creates Noise

The variance stems from several sources:

  1. Long credit assignment chains: Early actions get blamed (or credited) for everything that happens afterward, even random events
  2. Stochastic environments: Random outcomes multiply uncertainty across timesteps
  3. Stochastic policies: Random action sampling adds another layer of noise
  4. No bootstrapping: Unlike value-based methods, REINFORCE doesn't use learned estimates to reduce variance—it relies purely on actual sampled returns

Each gradient update in REINFORCE can point in wildly different directions because small environmental variations create large return differences. This means you need many samples to average out the noise, making learning painfully slow.

The Practical Impact

High variance manifests as:

  • Erratic training curves with huge fluctuations
  • Extremely slow convergence requiring millions of samples
  • Sensitivity to hyperparameters like learning rate
  • Risk of divergence if learning rate is too high

This is precisely why baseline subtraction (which you've already learned) is critical—it reduces variance without introducing bias.

Key Takeaway: REINFORCE's Monte Carlo returns create high-variance gradient estimates because they accumulate randomness across entire episodes without bootstrapping, requiring many samples to learn effectively and making baseline techniques essential.