REINFORCE Convergence Properties
What you'll learn: Why REINFORCE is mathematically guaranteed to find good policies and how your learning rate choice affects whether training stays stable or explodes.
The Convergence Guarantee
REINFORCE has a beautiful theoretical property: it's guaranteed to converge to a local optimum of the expected return. This comes from the Policy Gradient Theorem—since REINFORCE uses unbiased gradient estimates (Monte Carlo returns), it's following the true gradient direction on average. Think of it like climbing a hill in fog: even though each step might be slightly off due to noise, you're generally moving upward.
The key word is local optimum. Because neural network policies are non-convex (many hills and valleys), REINFORCE won't necessarily find the absolute best policy, but it will find a policy that's better than all nearby alternatives. This is still powerful—many RL tasks have good local optima.
Learning Rate: The Stability Knob
The learning rate is critical for REINFORCE's stability. Too large, and the policy changes so dramatically between episodes that it "forgets" what it learned—imagine taking giant leaps while hill-climbing and overshooting peaks. Too small, and learning crawls forward painfully slow, potentially getting stuck in bad regions.
REINFORCE is particularly sensitive because:
- High variance in gradient estimates amplifies learning rate problems
- Policy changes directly affect data distribution (on-policy learning)
- Large policy updates can cause catastrophic performance drops
Common practice uses adaptive optimizers (like Adam) with conservative initial learning rates (e.g., 1e-4 to 1e-3), and sometimes learning rate schedules that decay over time as the policy stabilizes.
Key Takeaway: REINFORCE converges to local optima in expectation due to unbiased gradients, but requires careful learning rate tuning to balance stability against learning speed—too high causes instability, too low causes slow convergence.