Course contentsShow
← Mathematics
Lesson 2289 of 3,12147. Linear Algebra — Applications and Advanced TopicsFree lesson

Principal Component Analysis (PCA) and SVD

Apply SVD to data matrices to perform PCA and extract principal components for dimensionality reduction.

Principal Component Analysis (PCA) and SVD

You'll learn how to use SVD on data matrices to extract the most important patterns and reduce dimensionality through PCA.

The Core Idea

Principal Component Analysis (PCA) is a technique for finding the "most important" directions in high-dimensional data. Imagine you have measurements of thousands of variables for each person in a study—PCA helps you discover that maybe just 3 or 4 underlying patterns explain most of the variation.

The connection to SVD is direct and powerful: when you perform SVD on a centered data matrix, the right singular vectors are exactly the principal components, and the singular values tell you how much variance each component captures.

How It Works

Given a data matrix X (rows = observations, columns = features):

  1. Center the data: Subtract the mean from each column to get X̃
  2. Compute SVD: X̃ = UΣVᵀ
  3. Extract components: The columns of V are your principal components
  4. Rank by importance: The singular values (diagonal of Σ) tell you the importance of each component
  5. Project and reduce: Transform data via X̃V and keep only the top k columns for dimensionality reduction

The first principal component points in the direction of maximum variance in your data. The second is orthogonal to the first and captures the next-most variance, and so on.

Analogy: If your data is a cloud of points in 3D space that looks like a flat pancake, PCA finds the plane the pancake lies on (2 components) and tells you the third dimension contributes almost nothing.

Key Takeaway: SVD transforms PCA from a conceptual idea into a concrete algorithm—apply SVD to centered data, and the right singular vectors immediately give you the principal components ranked by the singular values.