Low-Rank Approximation and Data Compression
You'll learn how to create the best possible low-rank approximation of a matrix by strategically discarding small singular values from the SVD.
The Core Idea
Once you have the Singular Value Decomposition (SVD) of a matrix A = UΣVᵀ, you can create a simpler, compressed version by keeping only the k largest singular values and setting the rest to zero. This gives you the best rank-k approximation of A in the Frobenius norm—no other rank-k matrix is closer to A.
Think of it like a photograph: the largest singular values capture the "main features" (major shapes, contrasts), while smaller singular values represent fine details or noise. By truncating small singular values, you keep the essence while discarding less important information, dramatically reducing storage requirements.
How It Works
Given A = UΣVᵀ where the singular values are ordered σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0:
- Choose k < r (the desired rank)
- Keep only the first k singular values in Σ, setting all others to zero
- Form Aₖ = UₖΣₖVₖᵀ, using only the first k columns of U and V
The matrix Aₖ is the closest rank-k matrix to A. The approximation error is exactly σₖ₊₁ (the first discarded singular value).
Why This Matters
This technique powers:
- Image compression: Store large images with fewer numbers
- Data analysis: Extract dominant patterns from noisy datasets
- Recommendation systems: Compress user-rating matrices
- Machine learning: Reduce dimensionality while preserving information
Key Takeaway: Truncating the SVD by keeping only the k largest singular values produces the optimal rank-k approximation of your matrix, enabling efficient data compression while minimizing information loss.