PCA Problem Formulation
You'll learn: How PCA mathematically defines the problem of finding the best directions to compress high-dimensional data.
The Core Mathematical Goal
You've learned that variance represents information in your data. Principal Component Analysis (PCA) takes this idea and asks: "Which directions in my data space capture the most variance?"
The formal objective of PCA is to find orthogonal (perpendicular) directions—called principal components—that maximize the variance of the data when projected onto them.
Why Orthogonal Directions?
Think of taking a 3D sculpture and shining a light to cast shadows on different walls. Each wall represents a direction of projection. PCA wants to find the wall angles where the shadow preserves the most detail (variance) from the original sculpture. Crucially, these "walls" must be perpendicular to each other to avoid redundancy—you want each direction to capture new information, not repeat what another direction already showed.
The Mathematical Setup
PCA seeks vectors (directions) such that:
- First component: The direction with maximum variance in the projected data
- Second component: The direction orthogonal to the first, with maximum remaining variance
- Third component: Orthogonal to both previous, with maximum remaining variance
- And so on...
This formulation ensures we rank dimensions by importance. If the first two components capture 95% of total variance, we can drop the rest with minimal information loss—achieving dimensionality reduction.
Key Takeaway: PCA formulates dimensionality reduction as an optimization problem: find orthogonal directions that sequentially maximize projected variance, allowing us to rank and select the most informative dimensions.