RMSprop: Exponential Averaging of Gradients
What You'll Learn
You'll learn how RMSprop fixes Adagrad's monotonic learning rate decay by using exponential moving averages instead of cumulative sums.
The Core Problem with Adagrad
Recall that Adagrad accumulates all past squared gradients to adapt learning rates. This means the denominator only grows larger, causing learning rates to shrink indefinitely. Eventually, the updates become so tiny that learning effectively stops, even if you haven't reached the optimal solution yet.
How RMSprop Solves This
RMSprop (Root Mean Square Propagation) replaces Adagrad's cumulative sum with an exponential moving average of squared gradients. Instead of remembering every gradient since training began, it gives more weight to recent gradients and "forgets" older ones gradually.
The key update equation:
v_t = β * v_{t-1} + (1 - β) * g_t²
θ_t = θ_{t-1} - (η / sqrt(v_t + ε)) * g_t
Where:
v_tis the exponentially weighted average of squared gradientsβis the decay rate (typically 0.9 or 0.99)g_tis the current gradientεis a small constant for numerical stability
The Key Difference
Think of it like tracking weather patterns: Adagrad keeps every temperature reading forever (accumulating endlessly), while RMSprop acts like a moving average that emphasizes recent days more heavily. This prevents the "learning paralysis" Adagrad suffers from.
With exponential averaging, the learning rate can both decrease and increase as needed, adapting to the current gradient landscape rather than monotonically declining.
Key Takeaway: RMSprop fixes Adagrad's aggressive learning rate decay by using exponential moving averages of squared gradients, allowing the optimizer to "forget" distant history and remain responsive throughout training.