Course contentsShow
← Machine Learning and Deep Learning
Lesson 713 of 3,53817. Optimization for Deep LearningFree lesson

Why Learning Rate Scheduling Matters

Understanding how fixed learning rates limit convergence and why dynamic schedules improve training efficiency and final performance.

Why Learning Rate Scheduling Matters

You'll learn: Why using a fixed learning rate throughout training can prevent your model from reaching its best possible performance, and how dynamic adjustment solves this.

The Problem with Fixed Learning Rates

Imagine learning to park a car. When you're far from the parking spot, you can drive quickly. But as you get closer, you need to slow down and make tiny adjustments—otherwise you'll overshoot and keep bouncing past the perfect position.

Training a neural network works the same way. Earlier in training, your weights are far from optimal, so a large learning rate helps you make rapid progress. But as you approach a good solution, that same large learning rate causes you to overshoot the minimum of your loss function, bouncing around without settling into the best weights.

Why Dynamic Schedules Improve Training

A learning rate schedule systematically reduces the learning rate as training progresses. This gives you:

  1. Fast initial progress: Start with a higher learning rate to quickly move toward good regions of the loss landscape
  2. Fine-tuned convergence: Gradually decrease the rate so your model can settle into a deeper, better minimum
  3. Better final performance: Avoid the oscillations that prevent a fixed rate from finding optimal weights

Think of it like zooming in on a map: you start with a wide view to find the general area, then zoom in for precise navigation.

Without scheduling, you face a dilemma: set the learning rate too high and you'll never converge; set it too low and training becomes painfully slow and may get stuck in poor local minima.

Key Takeaway: Learning rate scheduling bridges the gap between fast early training and precise late-stage convergence, allowing your model to reach better solutions than any single fixed learning rate could achieve.