Course contentsShow
← Machine Learning and Deep Learning
Lesson 702 of 3,53817. Optimization for Deep LearningFree lesson

AdaGrad: Per-Parameter Learning Rates

Understand how AdaGrad adapts learning rates based on historical gradient magnitudes for each parameter independently.

AdaGrad: Per-Parameter Learning Rates

What You'll Learn

You'll understand how AdaGrad automatically tunes learning rates for each individual parameter based on its history of gradient updates.

The Core Problem AdaGrad Solves

In standard SGD (and even momentum variants), every parameter shares the same learning rate. But imagine training a model where some features appear frequently while others are rare. Frequent features accumulate many gradient updates, while rare features get few. Using one global learning rate means frequent features might overshoot their optimal values while rare features barely move.

AdaGrad's innovation: Give each parameter its own adaptive learning rate that shrinks based on how much that parameter has been updated in the past.

How It Works

AdaGrad tracks the sum of squared gradients for each parameter separately:

  1. Accumulate gradient history: For each parameter, maintain a running sum of all its squared gradients
  2. Scale the learning rate: Divide the global learning rate by the square root of this accumulated sum
  3. Result: Parameters with large historical gradients get smaller effective learning rates; parameters with small historical gradients retain larger learning rates

Think of it like adjusting your car's speed based on how bumpy the road has been. If you've hit many potholes (large gradients), you slow down. If the road's been smooth (small gradients), you maintain speed.

The Mathematics

For each parameter, AdaGrad maintains an accumulator and updates:

  • Accumulator grows: sum of all past squared gradients
  • Effective learning rate shrinks: learning_rate / sqrt(accumulator + epsilon)

The epsilon prevents division by zero.

Key Takeaway: AdaGrad adapts learning rates per parameter by shrinking them proportionally to the accumulated magnitude of past gradients, giving rare features larger updates and frequent features smaller ones.