Feature Scaling for Gradient Descent
What You'll Learn
You'll understand why normalizing features speeds up gradient descent and prevents numerical problems during training.
The Problem with Unscaled Features
Imagine you're hiking down a valley where one direction is a gentle slope (kilometers wide) and another is a steep cliff (meters tall). Taking the same-sized step in each direction produces wildly different results—your progress becomes inefficient and erratic.
The same happens in gradient descent when features have different scales. If one feature ranges from 0-1 (like a proportion) and another from 0-100,000 (like house prices), the cost function's contour becomes elongated and narrow—like a stretched valley.
Why Scaling Accelerates Convergence
When features have similar scales:
- Balanced gradients: Each feature contributes proportionally to the gradient, so updates adjust all parameters sensibly
- Efficient learning rates: A single learning rate works well for all features—no need to move cautiously because one dimension dominates
- Straighter path: Gradient descent takes a more direct route toward the minimum instead of zig-zagging
Without scaling, gradient descent might oscillate wildly along dimensions with large-scale features while creeping slowly along small-scale ones.
Preventing Numerical Instability
Extremely large feature values can cause:
- Overflow: Computations exceed floating-point limits
- Exploding gradients: Parameter updates become massive and unpredictable
- Loss of precision: Small but important changes get rounded away
Feature scaling (like StandardScaler you've already seen) transforms all features to comparable ranges, typically with mean 0 and standard deviation 1.
Key Takeaway: Feature scaling makes gradient descent converge faster and more reliably by creating a well-conditioned optimization landscape where all features contribute equally, preventing the algorithm from wasting iterations bouncing around elongated cost function contours.