AdaGrad's Diminishing Learning Rate Problem
What You'll Learn
You'll understand why AdaGrad's clever per-parameter learning rates can become too small over time, effectively stopping your model from learning.
The Core Problem
Remember that AdaGrad adapts learning rates by accumulating the sum of all squared gradients from the beginning of training. The formula looks like:
α_t = α / sqrt(G_t + ε)
where G_t is the cumulative sum of squared gradients up to time t.
Here's the issue: this accumulator only grows—it never shrinks. With every training step, you're adding more squared gradient values to G_t, making the denominator larger and larger. As the denominator grows, your effective learning rate shrinks toward zero.
Why This Hurts Learning
Early in training, this aggressive decay can be helpful—it slows down parameters that are changing rapidly. But as training continues, even parameters that need significant updates get trapped with tiny learning rates. The model essentially "forgets" that it still needs to learn.
The Analogy
Imagine a car that permanently reduces its max speed every time you press the accelerator. At first, slowing down on highways makes sense. But eventually, even when you need to merge onto a new highway, your car can barely move—it remembers every acceleration from the past and won't let you speed up anymore.
When It Manifests
This problem is especially severe in:
- Long training runs
- Problems where optimal parameters keep shifting
- Deep networks with many parameters
Key Takeaway: AdaGrad's monotonically increasing gradient accumulator causes learning rates to decay too aggressively over time, often stopping learning prematurely before the model reaches optimal performance.