Adagrad: Adaptive Learning Rates
What You'll Learn
You'll learn how Adagrad automatically adapts the learning rate for each parameter based on its historical gradient information, solving the "one-size-fits-all" learning rate problem.
The Core Idea
Imagine training a model where some parameters (like rare feature weights) need large updates because they don't change often, while others (like common feature weights) need smaller, careful updates. Using a single learning rate for all parameters is inefficient.
Adagrad (Adaptive Gradient Algorithm) solves this by maintaining a running sum of squared gradients for each parameter. Parameters that have received large gradients in the past get smaller learning rates, while parameters with consistently small gradients get larger learning rates.
How It Works
For each parameter, Adagrad accumulates the sum of all squared gradients seen so far:
- Compute gradient
g_tat time stept - Add squared gradient to accumulator:
G_t = G_{t-1} + g_t² - Update parameter with adapted learning rate:
θ_t = θ_{t-1} - (η / √(G_t + ε)) * g_t
The denominator √(G_t + ε) grows over time for parameters with large gradients, effectively reducing their learning rate. The small constant ε (epsilon) prevents division by zero.
The Analogy
Think of it like adjusting your driving speed based on road familiarity. On roads you've traveled often (frequent large gradients), you slow down and make careful adjustments. On rarely-traveled roads (small historical gradients), you can move more boldly.
Key Limitation
Because G_t only grows (never shrinks), the learning rate continually decreases and can become infinitesimally small, potentially stopping learning prematurely in deep networks.
Key Takeaway: Adagrad adapts learning rates per parameter by dividing the base learning rate by the square root of accumulated squared gradients, enabling larger updates for infrequent parameters and smaller updates for frequent ones.