Course contentsShow
← Machine Learning and Deep Learning
Lesson 687 of 3,53817. Optimization for Deep LearningFree lesson

Learning Rate Too High or Too Low

Recognizing symptoms of poor learning rate choice: divergence, oscillation, and slow convergence.

Learning Rate Too High or Too Low

What You'll Learn

You'll learn to recognize the three classic symptoms that indicate your learning rate is poorly chosen: divergence, oscillation, and slow convergence.

The Three Warning Signs

The learning rate controls how big a step your optimizer takes when updating weights. Choose poorly, and training goes wrong in predictable ways.

1. Learning Rate Too High: Divergence

When your learning rate is too large, the optimizer takes huge jumps that overshoot the optimal weights. Your loss will explode upward or oscillate wildly, sometimes showing NaN (not a number) values. It's like trying to park a car by flooring the accelerator—you'll crash right past the parking spot.

Symptoms:

  • Loss increases instead of decreases
  • Loss becomes NaN or inf
  • Weights grow to extreme values

2. Learning Rate Too High: Oscillation

A moderately high learning rate causes the optimizer to bounce back and forth across the optimal point without settling. Your loss curve shows a noisy, zigzag pattern that never smoothly descends.

Symptoms:

  • Loss jumps up and down erratically
  • Training loss doesn't converge smoothly
  • Validation loss oscillates unpredictably

3. Learning Rate Too Low: Slow Convergence

When the learning rate is too small, training crawls forward at a snail's pace. The optimizer takes tiny steps, making progress imperceptibly slow—or getting stuck in local minima.

Symptoms:

  • Loss decreases very slowly
  • Training takes an impractically long time
  • Model barely improves after many epochs

The Goldilocks Principle

Finding the right learning rate is about balance: not too aggressive, not too timid, but just right for steady, efficient convergence.

Key Takeaway: A poorly chosen learning rate shows up as divergence (exploding loss), oscillation (erratic jumps), or slow convergence (minimal progress)—monitor your loss curves to diagnose and adjust accordingly.