Course contentsShow
← Machine Learning and Deep Learning
Lesson 714 of 3,53817. Optimization for Deep LearningFree lesson

Step Decay Schedules

Reducing learning rate by a fixed factor at predetermined epochs, balancing initial exploration with later fine-tuning.

Step Decay Schedules

What you'll learn: How to strategically reduce your learning rate at specific points during training to balance fast early progress with precise late-stage fine-tuning.

The Core Idea

Imagine learning to parallel park. At first, you make bold, large steering adjustments to get your car roughly into position. But as you get closer to perfect alignment, you switch to tiny, careful corrections. Step decay schedules apply this same logic to neural network training.

With step decay, you reduce your learning rate by a fixed factor (like dividing by 10) after a predetermined number of epochs. For example, you might start with a learning rate of 0.1, drop to 0.01 at epoch 30, then to 0.001 at epoch 60.

Why This Works

Early in training, a higher learning rate lets your optimizer explore the loss landscape quickly, making rapid progress toward good regions. But as training progresses, that same large learning rate can cause your model to "bounce around" the minimum rather than settling into it.

By stepping down the learning rate at strategic points, you allow:

  • Initial exploration: Big steps help escape poor local minima early
  • Later refinement: Smaller steps enable precise convergence to better solutions

Common Patterns

A typical step decay might reduce the learning rate by a factor of 10 every 30 epochs, or halve it every 20 epochs. The exact schedule depends on your dataset size, model architecture, and how quickly loss plateaus.

Key Takeaway: Step decay schedules strategically reduce learning rates at fixed intervals, letting your model explore boldly at first, then fine-tune carefully as training matures—like shifting from rough sketching to detailed brushwork.