The Problem SGD Variants Solve
What you'll learn: Why modern optimizers use per-parameter adaptive learning rates instead of a single global learning rate for all parameters.
The Challenge with Plain SGD
Imagine training a neural network where one feature represents income (ranging from 200,000) and another represents number of dependents (ranging from 0 to 5). When you use standard SGD with a single learning rate, you're applying the same update step size to parameters that process vastly different scales of data.
This creates two major problems:
1. Different Feature Scales
Parameters connected to large-scale features receive large gradients, while those connected to small-scale features receive tiny gradients. A learning rate that works well for one set of parameters might be catastrophically large or uselessly small for others. You end up either overshooting in some dimensions or crawling in others.
2. Sparse Gradients
In problems like natural language processing, most input features are zero most of the time (think of word embeddings where only a few words appear in each sentence). Parameters that rarely receive non-zero gradients learn extremely slowly with fixed learning rates, causing some parts of your network to lag far behind others.
Why This Matters
Without addressing these issues, you face:
- Slow convergence because you must use a conservatively small learning rate
- Poor final performance because some parameters never learn effectively
- Difficulty training deep networks where gradient magnitudes vary dramatically across layers
This is why momentum alone isn't enough—it smooths the direction but doesn't solve the scale mismatch problem. The next generation of optimizers (which you'll learn about soon) adapt the learning rate per parameter based on gradient history.
Key Takeaway: Per-parameter adaptive learning rates are essential because different parameters experience vastly different gradient scales and frequencies, making a single global learning rate inefficient or unstable.