Learning Rate Selection and Convergence
What you'll learn: How choosing the right learning rate determines whether gradient descent successfully finds the minimum, gets stuck oscillating, or explodes completely.
The Goldilocks Problem of Learning Rates
The learning rate (often denoted α or lr) is the step size in our gradient descent update rule. Remember from the previous lesson that we update our parameters by moving in the opposite direction of the gradient:
θ_new = θ_old - α * gradient
That α value is crucial—it controls how far you step down the cost function landscape.
Three Scenarios
Too Small: Slow Convergence
If your learning rate is too small, gradient descent takes tiny steps. It will eventually reach the minimum, but painfully slowly—potentially requiring thousands or millions of iterations. Think of taking baby steps down a mountain when you could be walking normally.
Too Large: Oscillation or Divergence
If your learning rate is too large, you overshoot the minimum. Instead of gradually descending into the valley, you bounce from one side to the other, never settling down (oscillation). Even worse, steps so large can cause you to jump further from the minimum with each iteration (divergence)—your loss actually increases!
Just Right: Efficient Convergence
The ideal learning rate takes reasonably sized steps that consistently decrease the loss, converging to the minimum in a reasonable number of iterations without wild oscillations.
How to Tell
Monitor your loss curve during training. A well-chosen learning rate shows smooth, steady decrease. Oscillating loss suggests it's too high. A barely-changing loss suggests it's too low.
Key Takeaway: The learning rate controls convergence behavior—too small means slow progress, too large causes oscillation or divergence, and finding the right balance is essential for effective gradient descent training.