Course contentsShow
← Machine Learning and Deep Learning
Lesson 691 of 3,53817. Optimization for Deep LearningFree lesson

The Problem SGD Variants Solve

Why per-parameter adaptive learning rates are needed for different feature scales and sparse gradients.

The Problem SGD Variants Solve

What you'll learn: Why modern optimizers use per-parameter adaptive learning rates instead of a single global learning rate for all parameters.

The Challenge with Plain SGD

Imagine training a neural network where one feature represents income (ranging from 20,000to20,000 to 200,000) and another represents number of dependents (ranging from 0 to 5). When you use standard SGD with a single learning rate, you're applying the same update step size to parameters that process vastly different scales of data.

This creates two major problems:

1. Different Feature Scales

Parameters connected to large-scale features receive large gradients, while those connected to small-scale features receive tiny gradients. A learning rate that works well for one set of parameters might be catastrophically large or uselessly small for others. You end up either overshooting in some dimensions or crawling in others.

2. Sparse Gradients

In problems like natural language processing, most input features are zero most of the time (think of word embeddings where only a few words appear in each sentence). Parameters that rarely receive non-zero gradients learn extremely slowly with fixed learning rates, causing some parts of your network to lag far behind others.

Why This Matters

Without addressing these issues, you face:

  • Slow convergence because you must use a conservatively small learning rate
  • Poor final performance because some parameters never learn effectively
  • Difficulty training deep networks where gradient magnitudes vary dramatically across layers

This is why momentum alone isn't enough—it smooths the direction but doesn't solve the scale mismatch problem. The next generation of optimizers (which you'll learn about soon) adapt the learning rate per parameter based on gradient history.

Key Takeaway: Per-parameter adaptive learning rates are essential because different parameters experience vastly different gradient scales and frequencies, making a single global learning rate inefficient or unstable.