Course contentsShow
← Machine Learning and Deep Learning
Lesson 692 of 3,53817. Optimization for Deep LearningFree lesson

Adagrad: Adaptive Learning Rates

Accumulating squared gradients to scale learning rates inversely with past gradient magnitudes per parameter.

Adagrad: Adaptive Learning Rates

What You'll Learn

You'll learn how Adagrad automatically adapts the learning rate for each parameter based on its historical gradient information, solving the "one-size-fits-all" learning rate problem.

The Core Idea

Imagine training a model where some parameters (like rare feature weights) need large updates because they don't change often, while others (like common feature weights) need smaller, careful updates. Using a single learning rate for all parameters is inefficient.

Adagrad (Adaptive Gradient Algorithm) solves this by maintaining a running sum of squared gradients for each parameter. Parameters that have received large gradients in the past get smaller learning rates, while parameters with consistently small gradients get larger learning rates.

How It Works

For each parameter, Adagrad accumulates the sum of all squared gradients seen so far:

  1. Compute gradient g_t at time step t
  2. Add squared gradient to accumulator: G_t = G_{t-1} + g_t²
  3. Update parameter with adapted learning rate: θ_t = θ_{t-1} - (η / √(G_t + ε)) * g_t

The denominator √(G_t + ε) grows over time for parameters with large gradients, effectively reducing their learning rate. The small constant ε (epsilon) prevents division by zero.

The Analogy

Think of it like adjusting your driving speed based on road familiarity. On roads you've traveled often (frequent large gradients), you slow down and make careful adjustments. On rarely-traveled roads (small historical gradients), you can move more boldly.

Key Limitation

Because G_t only grows (never shrinks), the learning rate continually decreases and can become infinitesimally small, potentially stopping learning prematurely in deep networks.

Key Takeaway: Adagrad adapts learning rates per parameter by dividing the base learning rate by the square root of accumulated squared gradients, enabling larger updates for infrequent parameters and smaller updates for frequent ones.