Why Fixed Learning Rates Fail
What you'll learn: Why using one constant learning rate throughout training often leads to suboptimal results and unstable convergence.
The Core Problem
Imagine trying to navigate a mountain landscape wearing boots that force you to take exactly the same stride length no matter the terrain. On gentle slopes, you'd move painfully slowly. On steep cliffs, you'd overshoot safe paths and possibly fall.
Fixed learning rates create the same problem in neural networks. A constant learning rate treats all parameters equally and assumes the loss landscape doesn't change — but both assumptions are wrong.
Why One Size Doesn't Fit All
Different parameters need different magnitudes: Some weights connect to frequently-activated neurons (think common words in text), while others fire rarely (rare words). Using the same learning rate means common features get updated with massive, noisy gradients, while rare features barely move.
The landscape changes during training: Early in training, you're far from the optimum and can take large steps. Near convergence, the loss surface becomes a narrow valley — the same large step now bounces you between walls instead of moving forward. A fixed rate either gets stuck early or oscillates wildly later.
Layer depth matters: In deep networks (as we saw with gradient flow problems), early layers receive smaller gradients than later layers. A learning rate that works for the output layer may be far too aggressive for input layers.
The Real-World Consequence
With a fixed learning rate, you're forced to choose a compromise value that's never quite right for any part of your network or any stage of training. You sacrifice either early-stage learning speed or late-stage precision.
Key Takeaway: Fixed learning rates fail because neural networks have heterogeneous parameters with time-varying gradient magnitudes — modern optimizers solve this with per-parameter adaptive rates.