Variance as Information
What you'll learn: Why high-variance directions in your data are considered "signal" worth keeping, while low-variance directions often represent noise or redundancy.
The Core Idea
Imagine filming a busy street from above. Most of the action—cars moving, people walking—happens along the road (high variance). Very little happens in the perpendicular direction across narrow sidewalks (low variance). If you had to describe this scene in fewer words, you'd focus on the road direction because that's where the information is.
In dimensionality reduction, variance equals information. Features or directions along which your data varies greatly typically capture meaningful patterns, differences between samples, and the structure you care about. Directions with little variance contain mostly redundant information or measurement noise—they don't help distinguish one data point from another.
Why This Matters
When you have high-dimensional data (many features), some dimensions contribute more to understanding your dataset than others. A feature where all values are nearly identical (low variance) tells you almost nothing useful. Meanwhile, a feature with high variance reveals how samples differ from each other—that's the "signal" you want to preserve.
Think of it like audio: the loud, varying sounds are the music (high variance = information). The quiet, constant hum is just background noise (low variance = discard it).
Dimensionality reduction techniques prioritize keeping directions with high variance because losing them means losing the patterns that make your data interesting and useful for downstream tasks like classification or clustering.
Key Takeaway: Variance represents information content—high-variance directions capture meaningful signal, while low-variance directions often contain noise or redundancy that can be safely discarded during dimensionality reduction.