Course contentsShow
← Machine Learning and Deep Learning
Lesson 686 of 3,53817. Optimization for Deep LearningFree lesson

The Learning Rate: Core Hyperparameter

Why learning rate is the most critical hyperparameter and how it controls the step size in parameter space.

The Learning Rate: Core Hyperparameter

What you'll learn: Why the learning rate is the single most important hyperparameter in neural network training and how it controls optimization behavior.

What Is the Learning Rate?

The learning rate determines how big a step your optimizer takes in parameter space when updating weights. After computing gradients through backpropagation, you don't just subtract the raw gradient from your weights—you multiply it by the learning rate first:

new_weight = old_weight - learning_rate × gradient

Why It's the Most Critical Hyperparameter

Think of training as hiking down a mountain in fog (your loss landscape). The gradient tells you which direction is downhill, but the learning rate decides how far you step:

  • Too large: You take giant leaps and might jump right over the valley bottom, or even bounce uphill. Your loss oscillates wildly or explodes.
  • Too small: You take tiny, cautious steps. Training is painfully slow and you might get stuck in shallow local minima before reaching better solutions.
  • Just right: You make steady progress toward lower loss without overshooting.

The Balancing Act

Unlike batch size or network architecture, the learning rate directly controls the trade-off between:

  • Convergence speed (how fast you reach good performance)
  • Training stability (whether your loss decreases smoothly)
  • Final performance (whether you settle into a good minimum)

A poor learning rate can make even a perfect architecture fail to learn anything at all. Conversely, a well-tuned learning rate can salvage suboptimal choices elsewhere.

Key Takeaway: The learning rate is the master control knob for optimization—it directly scales every parameter update and determines whether your network learns effectively, slowly, or not at all.