Course contentsShow
← Computer Science
Lesson 2821 of 2,87263. Classical ML AlgorithmsFree lesson

Learning Rate and Convergence

Understanding how step size affects training speed and stability in gradient descent optimization.

Learning Rate and Convergence

You'll learn how the step size in gradient descent controls both how quickly you reach a solution and whether you reach one at all.

The Core Idea

When performing gradient descent, the learning rate (often denoted α or η) determines how big a step you take in the direction opposite to the gradient. Think of it like adjusting your stride while hiking down a foggy mountain—you can't see the bottom, only which way is downhill right where you stand.

Why Learning Rate Matters

Too small: You take tiny, careful steps. You'll eventually reach the valley bottom (convergence), but it might take thousands or millions of iterations. Training becomes painfully slow.

Too large: You take giant leaps and might overshoot the minimum entirely. You could bounce back and forth across the valley, or even jump to worse positions uphill. Your loss might oscillate wildly or even increase—you fail to converge.

Just right: You make steady progress toward the minimum in a reasonable number of iterations, balancing speed with stability.

The Convergence Challenge

Convergence means your algorithm successfully reaches a point where gradients are near zero (a minimum). With gradient descent for linear regression, the update rule is:

θ = θ - α * gradient

The learning rate α directly multiplies your correction. If α is poorly chosen, you might see:

  • Slow, steady improvement (too small)
  • Erratic jumps in loss (too large)
  • Perfect, smooth descent curves (well-tuned)

In practice, you often start with common values like 0.01, 0.001, or 0.0001 and adjust based on observing your loss function over iterations.

Key Takeaway: The learning rate is a critical hyperparameter that balances training speed against stability—too small wastes time, too large prevents convergence, and finding the sweet spot requires experimentation.