Course contentsShow
← Machine Learning and Deep Learning
Lesson 101 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Learning Rate and Step Size

How step size affects convergence speed and stability; choosing and tuning the learning rate.

Learning Rate and Step Size

You'll learn how the learning rate controls how quickly (and safely) your model learns during optimization.

What Is Learning Rate?

In gradient descent (the algorithm you just learned), the learning rate (often denoted α or η) determines how big a step you take in the direction opposite the gradient. Think of it like adjusting your stride when walking downhill in fog—too small and you'll take forever to reach the bottom; too large and you might overshoot or stumble off course entirely.

Mathematically, when updating parameters:

θ_new = θ_old - α × gradient

The learning rate α multiplies the gradient, scaling your step size.

Finding the Sweet Spot

Too small: Your model learns very slowly. Gradient descent takes tiny steps, requiring thousands or millions of iterations to converge. Training becomes painfully slow.

Too large: You risk instability. Large steps can overshoot the minimum entirely, bouncing around the optimal point or even diverging (getting worse over time). Your loss function might oscillate wildly or explode.

Just right: The model converges efficiently—fast enough to be practical, stable enough to reliably find a good minimum.

Tuning Strategy

Common starting values range from 0.001 to 0.1. You typically:

  • Start with a moderate value (e.g., 0.01)
  • Monitor your loss function during training
  • Decrease it if training is unstable
  • Increase it if progress is too slow

Some advanced techniques adjust the learning rate automatically during training, but the core principle remains: balance speed against stability.

Key Takeaway: The learning rate is the most critical hyperparameter in gradient descent—it controls the trade-off between convergence speed and training stability.