Learning Rate Too High or Too Low
What You'll Learn
You'll learn to recognize the three classic symptoms that indicate your learning rate is poorly chosen: divergence, oscillation, and slow convergence.
The Three Warning Signs
The learning rate controls how big a step your optimizer takes when updating weights. Choose poorly, and training goes wrong in predictable ways.
1. Learning Rate Too High: Divergence
When your learning rate is too large, the optimizer takes huge jumps that overshoot the optimal weights. Your loss will explode upward or oscillate wildly, sometimes showing NaN (not a number) values. It's like trying to park a car by flooring the accelerator—you'll crash right past the parking spot.
Symptoms:
- Loss increases instead of decreases
- Loss becomes
NaNorinf - Weights grow to extreme values
2. Learning Rate Too High: Oscillation
A moderately high learning rate causes the optimizer to bounce back and forth across the optimal point without settling. Your loss curve shows a noisy, zigzag pattern that never smoothly descends.
Symptoms:
- Loss jumps up and down erratically
- Training loss doesn't converge smoothly
- Validation loss oscillates unpredictably
3. Learning Rate Too Low: Slow Convergence
When the learning rate is too small, training crawls forward at a snail's pace. The optimizer takes tiny steps, making progress imperceptibly slow—or getting stuck in local minima.
Symptoms:
- Loss decreases very slowly
- Training takes an impractically long time
- Model barely improves after many epochs
The Goldilocks Principle
Finding the right learning rate is about balance: not too aggressive, not too timid, but just right for steady, efficient convergence.
Key Takeaway: A poorly chosen learning rate shows up as divergence (exploding loss), oscillation (erratic jumps), or slow convergence (minimal progress)—monitor your loss curves to diagnose and adjust accordingly.