Course contentsShow
← Machine Learning and Deep Learning
Lesson 2272 of 3,53849. Deep Reinforcement Learning: Policy-BasedFree lesson

REINFORCE Convergence Properties

Why REINFORCE is guaranteed to converge to local optima and how learning rate affects stability.

REINFORCE Convergence Properties

What you'll learn: Why REINFORCE is mathematically guaranteed to find good policies and how your learning rate choice affects whether training stays stable or explodes.

The Convergence Guarantee

REINFORCE has a beautiful theoretical property: it's guaranteed to converge to a local optimum of the expected return. This comes from the Policy Gradient Theorem—since REINFORCE uses unbiased gradient estimates (Monte Carlo returns), it's following the true gradient direction on average. Think of it like climbing a hill in fog: even though each step might be slightly off due to noise, you're generally moving upward.

The key word is local optimum. Because neural network policies are non-convex (many hills and valleys), REINFORCE won't necessarily find the absolute best policy, but it will find a policy that's better than all nearby alternatives. This is still powerful—many RL tasks have good local optima.

Learning Rate: The Stability Knob

The learning rate is critical for REINFORCE's stability. Too large, and the policy changes so dramatically between episodes that it "forgets" what it learned—imagine taking giant leaps while hill-climbing and overshooting peaks. Too small, and learning crawls forward painfully slow, potentially getting stuck in bad regions.

REINFORCE is particularly sensitive because:

  • High variance in gradient estimates amplifies learning rate problems
  • Policy changes directly affect data distribution (on-policy learning)
  • Large policy updates can cause catastrophic performance drops

Common practice uses adaptive optimizers (like Adam) with conservative initial learning rates (e.g., 1e-4 to 1e-3), and sometimes learning rate schedules that decay over time as the policy stabilizes.

Key Takeaway: REINFORCE converges to local optima in expectation due to unbiased gradients, but requires careful learning rate tuning to balance stability against learning speed—too high causes instability, too low causes slow convergence.