Course contentsShow
← Machine Learning and Deep Learning
Lesson 24 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Matrix Approximation with SVD

Master low-rank matrix approximation using truncated SVD for dimensionality reduction.

Matrix Approximation with SVD

You'll learn: How to compress large matrices by keeping only their most important information using truncated SVD.

The Core Idea

Imagine you have a huge dataset stored as a matrix—maybe thousands of rows and columns. Singular Value Decomposition (SVD) breaks this matrix into three components: U, Σ (sigma, the diagonal matrix of singular values), and V^T. Here's the magic: the singular values in Σ are sorted from largest to smallest, representing importance.

Low-rank approximation means we keep only the top k singular values and their corresponding columns/rows from U and V^T, then reconstruct an approximate version of the original matrix. This is called truncated SVD.

Why It Matters

Think of a high-resolution photograph. Full SVD is like storing every pixel perfectly. Truncated SVD is like a smart compression: you keep the dominant patterns (large singular values) and discard tiny details (small singular values). The result? A smaller, faster representation that captures most of the original information.

Mathematically, if A is your original m × n matrix and you keep only k singular values:

A ≈ A_k = U_k Σ_k V_k^T

where U_k is m × k, Σ_k is k × k, and V_k^T is k × n. You've reduced the rank from potentially min(m,n) down to k, dramatically reducing storage and computation.

The Benefit

This technique is foundational for dimensionality reduction. Instead of working with thousands of features, you work with k "compressed" dimensions that preserve the most variance in your data. The larger the singular values you keep, the better your approximation.

Key Takeaway: Truncated SVD lets you approximate any matrix by keeping only its top-k singular values, achieving efficient dimensionality reduction while preserving the most important information.