RMSprop: Exponential Moving Average of Gradients
What You'll Learn
You'll discover how RMSprop fixes AdaGrad's shrinking learning rate problem by using exponential decay to keep recent gradients most relevant.
The Problem with AdaGrad
Remember how AdaGrad accumulated all past squared gradients? This caused the learning rate to shrink indefinitely—eventually grinding learning to a halt, especially in non-convex problems where the landscape keeps changing.
How RMSprop Solves It
RMSprop (Root Mean Square Propagation) uses an exponential moving average instead of summing all gradients forever. Think of it like a sliding window that gives more weight to recent gradients and lets old ones fade away.
Instead of:
sum_of_squares = sum_of_squares + gradient² # grows forever
RMSprop does:
moving_avg = decay_rate * moving_avg + (1 - decay_rate) * gradient²
The decay_rate (typically 0.9 or 0.99) controls the "memory length." With decay_rate=0.9, recent gradients matter most, while old gradients fade exponentially.
The Analogy
Imagine tracking temperature to decide what to wear. AdaGrad is like averaging every temperature since you were born—ancient cold winters would still convince you to wear a coat in summer! RMSprop is like checking the recent week's temperatures—much more relevant to today's decision.
The Update Rule
moving_avg_t = β * moving_avg_(t-1) + (1 - β) * gradient²
learning_rate_adjusted = learning_rate / sqrt(moving_avg_t + ε)
parameter = parameter - learning_rate_adjusted * gradient
Where β is the decay rate (usually 0.9) and ε prevents division by zero.
Key Takeaway: RMSprop prevents AdaGrad's vanishing learning rates by replacing infinite accumulation with exponential moving averages, keeping adaptation responsive to recent gradient behavior while forgetting distant past.