Course contentsShow
← Machine Learning and Deep Learning
Lesson 385 of 3,5389. Dimensionality ReductionFree lesson

PCA Problem Formulation

Formal objective: finding orthogonal directions that maximize projected variance in the data.

PCA Problem Formulation

You'll learn: How PCA mathematically defines the problem of finding the best directions to compress high-dimensional data.

The Core Mathematical Goal

You've learned that variance represents information in your data. Principal Component Analysis (PCA) takes this idea and asks: "Which directions in my data space capture the most variance?"

The formal objective of PCA is to find orthogonal (perpendicular) directions—called principal components—that maximize the variance of the data when projected onto them.

Why Orthogonal Directions?

Think of taking a 3D sculpture and shining a light to cast shadows on different walls. Each wall represents a direction of projection. PCA wants to find the wall angles where the shadow preserves the most detail (variance) from the original sculpture. Crucially, these "walls" must be perpendicular to each other to avoid redundancy—you want each direction to capture new information, not repeat what another direction already showed.

The Mathematical Setup

PCA seeks vectors (directions) such that:

  1. First component: The direction with maximum variance in the projected data
  2. Second component: The direction orthogonal to the first, with maximum remaining variance
  3. Third component: Orthogonal to both previous, with maximum remaining variance
  4. And so on...

This formulation ensures we rank dimensions by importance. If the first two components capture 95% of total variance, we can drop the rest with minimal information loss—achieving dimensionality reduction.

Key Takeaway: PCA formulates dimensionality reduction as an optimization problem: find orthogonal directions that sequentially maximize projected variance, allowing us to rank and select the most informative dimensions.