Standardization Before PCA
What You'll Learn
You'll understand why standardizing features before PCA prevents variables with large scales from unfairly dominating your principal components.
Why Feature Scaling Matters for PCA
Remember that PCA finds directions of maximum variance in your data. Here's the problem: variance is measured in the original units of each feature.
Imagine you have a dataset with two features:
- House price (ranging from 1,000,000)
- Number of bedrooms (ranging from 1 to 5)
The variance of house prices will be astronomically larger than the variance of bedrooms—not because price is more important, but simply because the numbers are bigger! When PCA searches for the direction of maximum variance, it will focus almost entirely on the house price feature and ignore bedrooms.
The Solution: Standardization
By standardizing your features (subtracting the mean and dividing by standard deviation), you transform all features to have:
- Mean = 0
- Standard deviation = 1
Now both features live on the same scale. PCA can fairly assess which directions capture the most meaningful variation in the data, rather than being misled by arbitrary units (dollars vs. counts).
The Analogy
Think of it like comparing noise levels. One person shouts their opinion in decibels, another whispers theirs in millibascal units. Even if both opinions are equally important, the shouter dominates the conversation—not because they're right, but because they're louder. Standardization gives everyone the same volume.
Key Takeaway: Always standardize your features before PCA unless you have a specific reason to preserve the original scale, because features with large variance will otherwise dominate the principal components regardless of their actual importance.