Course contentsShow
← Machine Learning and Deep Learning
Lesson 219 of 3,5383. Linear RegressionFree lesson

Feature Scaling for Gradient Descent

Understand why normalizing features accelerates convergence and prevents numerical instability in gradient descent.

Feature Scaling for Gradient Descent

What You'll Learn

You'll understand why normalizing features speeds up gradient descent and prevents numerical problems during training.

The Problem with Unscaled Features

Imagine you're hiking down a valley where one direction is a gentle slope (kilometers wide) and another is a steep cliff (meters tall). Taking the same-sized step in each direction produces wildly different results—your progress becomes inefficient and erratic.

The same happens in gradient descent when features have different scales. If one feature ranges from 0-1 (like a proportion) and another from 0-100,000 (like house prices), the cost function's contour becomes elongated and narrow—like a stretched valley.

Why Scaling Accelerates Convergence

When features have similar scales:

  1. Balanced gradients: Each feature contributes proportionally to the gradient, so updates adjust all parameters sensibly
  2. Efficient learning rates: A single learning rate works well for all features—no need to move cautiously because one dimension dominates
  3. Straighter path: Gradient descent takes a more direct route toward the minimum instead of zig-zagging

Without scaling, gradient descent might oscillate wildly along dimensions with large-scale features while creeping slowly along small-scale ones.

Preventing Numerical Instability

Extremely large feature values can cause:

  • Overflow: Computations exceed floating-point limits
  • Exploding gradients: Parameter updates become massive and unpredictable
  • Loss of precision: Small but important changes get rounded away

Feature scaling (like StandardScaler you've already seen) transforms all features to comparable ranges, typically with mean 0 and standard deviation 1.

Key Takeaway: Feature scaling makes gradient descent converge faster and more reliably by creating a well-conditioned optimization landscape where all features contribute equally, preventing the algorithm from wasting iterations bouncing around elongated cost function contours.