Course contentsShow
← Machine Learning and Deep Learning
Lesson 694 of 3,53817. Optimization for Deep LearningFree lesson

RMSprop: Exponential Averaging of Gradients

Using exponential moving average of squared gradients to prevent Adagrad's monotonic decay problem.

RMSprop: Exponential Averaging of Gradients

What You'll Learn

You'll learn how RMSprop fixes Adagrad's monotonic learning rate decay by using exponential moving averages instead of cumulative sums.

The Core Problem with Adagrad

Recall that Adagrad accumulates all past squared gradients to adapt learning rates. This means the denominator only grows larger, causing learning rates to shrink indefinitely. Eventually, the updates become so tiny that learning effectively stops, even if you haven't reached the optimal solution yet.

How RMSprop Solves This

RMSprop (Root Mean Square Propagation) replaces Adagrad's cumulative sum with an exponential moving average of squared gradients. Instead of remembering every gradient since training began, it gives more weight to recent gradients and "forgets" older ones gradually.

The key update equation:

v_t = β * v_{t-1} + (1 - β) * g_t²
θ_t = θ_{t-1} - (η / sqrt(v_t + ε)) * g_t

Where:

  • v_t is the exponentially weighted average of squared gradients
  • β is the decay rate (typically 0.9 or 0.99)
  • g_t is the current gradient
  • ε is a small constant for numerical stability

The Key Difference

Think of it like tracking weather patterns: Adagrad keeps every temperature reading forever (accumulating endlessly), while RMSprop acts like a moving average that emphasizes recent days more heavily. This prevents the "learning paralysis" Adagrad suffers from.

With exponential averaging, the learning rate can both decrease and increase as needed, adapting to the current gradient landscape rather than monotonically declining.

Key Takeaway: RMSprop fixes Adagrad's aggressive learning rate decay by using exponential moving averages of squared gradients, allowing the optimizer to "forget" distant history and remain responsive throughout training.