Feature Scaling and Normalization
You'll learn why preprocessing inputs to similar numerical ranges accelerates training and prevents numerical instability in machine learning models.
The Core Problem
Imagine you're training a linear regression model with two features: house size (in square feet, ranging 500–5000) and number of bedrooms (ranging 1–5). Without scaling, your gradient descent algorithm treats a 100-unit change in square footage the same mathematically as a 100-unit change in bedrooms—but one is tiny, the other impossible!
Features with larger magnitudes dominate the gradient updates, causing slow, zigzagging convergence. The learning rate that works for large-scale features might overshoot small-scale ones, or vice versa.
Why Scale Features?
Faster convergence: When all features live in similar ranges (like 0–1 or standard deviations from the mean), gradient descent takes more direct paths toward the minimum. The loss landscape becomes spherical rather than elliptical.
Numerical stability: Extreme value differences can cause floating-point overflow or underflow, especially in deep models with many layers of computation.
Fair feature treatment: Algorithms using distance metrics (like k-nearest neighbors) or regularization (L1/L2) treat all features equitably when scaled.
Common Techniques
Min-Max Normalization (rescales to [0,1]):
X_scaled = (X - X_min) / (X_max - X_min)
Standardization (zero mean, unit variance):
X_scaled = (X - mean) / std_dev
Use min-max when you need bounded ranges; use standardization when your data has outliers or you want to preserve the distribution shape.
Key Takeaway: Feature scaling preprocesses inputs to comparable ranges, dramatically improving gradient descent efficiency and numerical stability—essential before training most classical ML models.