Adagrad's Limitations
What You'll Learn
You'll discover why Adagrad's aggressive learning rate decay can cause training to stop too early, motivating the need for better adaptive optimization methods.
The Problem with Adagrad's Accumulation
Recall that Adagrad adapts learning rates by dividing by the square root of accumulated squared gradients. While this helps parameters with large gradients take smaller steps and vice versa, there's a critical flaw in the design.
The accumulator never forgets. Adagrad continuously adds to its gradient history throughout training, meaning the denominator grows monotonically—it only gets larger, never smaller.
Why This Causes Premature Convergence
As training progresses, the accumulated sum of squared gradients becomes very large. This makes the effective learning rate:
effective_lr = initial_lr / sqrt(accumulated_sum)
shrink toward zero. Eventually, the learning rate becomes so tiny that parameter updates are essentially negligible—training effectively stops, even if you haven't reached the optimal solution yet.
The Early Stopping Analogy
Think of Adagrad like a car that remembers every mile driven and gradually reduces its maximum speed accordingly. At first, this adaptive slowdown is helpful for smooth navigation. But eventually, the car becomes so slow from accumulated mileage that it can barely move forward, even on a clear highway where faster speeds would be safe and beneficial.
This aggressive decay is particularly problematic in deep learning, where training often requires many iterations through complex, non-convex loss landscapes.
The Motivation for Better Methods
Adagrad's premature convergence limitation directly motivated improvements like RMSprop and Adam, which use exponential moving averages instead of full accumulation to maintain more stable, usable learning rates throughout training.
Key Takeaway: Adagrad's unlimited accumulation of squared gradients causes learning rates to decay too aggressively, leading to premature training stoppage and motivating adaptive methods with better long-term behavior.