Course contentsShow
← Machine Learning and Deep Learning
Lesson 407 of 3,53810. Feature EngineeringFree lesson

Why Feature Scaling Matters

Understanding how different feature scales affect distance-based algorithms, gradient descent convergence, and regularization.

Why Feature Scaling Matters

What you'll learn: How differences in feature magnitude can sabotage your models and why bringing features to a common scale is essential for many algorithms.

The Problem: Features with Different Scales

Imagine you're building a model to predict house prices using two features: square footage (ranging from 500 to 5000) and number of bedrooms (ranging from 1 to 5). The square footage values are 1000 times larger than bedroom counts!

Many machine learning algorithms are sensitive to these magnitude differences. When features operate at wildly different scales, it creates three critical problems:

1. Distance-Based Algorithms Get Confused

Algorithms like K-Nearest Neighbors and K-Means clustering rely on measuring distances between data points. When one feature has much larger values, it dominates the distance calculation. In our example, the algorithm would essentially ignore bedroom count and make decisions based almost entirely on square footage—even if bedrooms are equally important.

2. Gradient Descent Struggles to Converge

Training models like logistic regression or neural networks uses gradient descent optimization. Unscaled features create elongated, narrow cost function landscapes that gradient descent must navigate. This leads to slow, zigzagging convergence or getting stuck entirely. With scaled features, the path to the minimum becomes more direct.

3. Regularization Becomes Unfair

Regularization techniques (L1/L2) penalize large coefficients. But when features have different scales, coefficients naturally differ in magnitude just to compensate for their inputs. A feature with values 0–1 needs a large coefficient; a feature with values 0–10000 needs a tiny one. Regularization then unfairly penalizes the large coefficient, even though both features might be equally important.

Key Takeaway: Feature scaling ensures algorithms treat all features fairly by bringing them to comparable ranges, enabling faster convergence, better distance calculations, and fairer regularization.