AdaGrad: Per-Parameter Learning Rates
What You'll Learn
You'll understand how AdaGrad automatically tunes learning rates for each individual parameter based on its history of gradient updates.
The Core Problem AdaGrad Solves
In standard SGD (and even momentum variants), every parameter shares the same learning rate. But imagine training a model where some features appear frequently while others are rare. Frequent features accumulate many gradient updates, while rare features get few. Using one global learning rate means frequent features might overshoot their optimal values while rare features barely move.
AdaGrad's innovation: Give each parameter its own adaptive learning rate that shrinks based on how much that parameter has been updated in the past.
How It Works
AdaGrad tracks the sum of squared gradients for each parameter separately:
- Accumulate gradient history: For each parameter, maintain a running sum of all its squared gradients
- Scale the learning rate: Divide the global learning rate by the square root of this accumulated sum
- Result: Parameters with large historical gradients get smaller effective learning rates; parameters with small historical gradients retain larger learning rates
Think of it like adjusting your car's speed based on how bumpy the road has been. If you've hit many potholes (large gradients), you slow down. If the road's been smooth (small gradients), you maintain speed.
The Mathematics
For each parameter, AdaGrad maintains an accumulator and updates:
- Accumulator grows: sum of all past squared gradients
- Effective learning rate shrinks:
learning_rate / sqrt(accumulator + epsilon)
The epsilon prevents division by zero.
Key Takeaway: AdaGrad adapts learning rates per parameter by shrinking them proportionally to the accumulated magnitude of past gradients, giving rare features larger updates and frequent features smaller ones.