Computing and Interpreting SVD
What you'll learn: How to break down matrices using SVD and understand what those singular values actually mean.
The Mechanics of SVD
You already know that Singular Value Decomposition (SVD) factorizes any matrix into three components. Now let's see how it works in practice.
For any matrix A (m×n), SVD gives you: A = UΣV^T
Where:
- U (m×m): Orthogonal matrix whose columns are left singular vectors (directions in output space)
- Σ (m×n): Diagonal matrix containing singular values (σ₁ ≥ σ₂ ≥ ... ≥ 0)
- V^T (n×n): Orthogonal matrix whose rows are right singular vectors (directions in input space)
What Singular Values Tell You
Think of singular values as importance scores for different directions in your data. The largest singular value (σ₁) corresponds to the direction of maximum variance—the most important pattern. The second largest (σ₂) captures the next most important pattern, orthogonal to the first, and so on.
Analogy: Imagine compressing a photo. The largest singular values capture the essential structure (edges, main shapes), while tiny singular values represent fine details or noise. Drop the smallest values, and you still recognize the image—that's dimensionality reduction in action!
Interpreting the Decomposition
- Large singular values → Important directions that capture significant variation
- Small singular values → Less important directions, possibly noise
- Zero singular values → Dimensions that contribute nothing (related to rank)
The rank of matrix A equals the number of non-zero singular values—connecting back to what you learned about matrix rank and column space.
Key Takeaway: SVD computation reveals hidden structure by ordering directions from most to least important, with singular values quantifying each direction's significance for representing your data.