What is the Singular Value Decomposition?
You'll learn: How any matrix can be factorized into three special matrices that reveal its fundamental geometric structure.
The Core Idea
The Singular Value Decomposition (SVD) is a factorization that breaks any matrix A (m×n) into three components:
A = UΣV^T
where:
- U is an m×m orthogonal matrix (columns are orthonormal eigenvectors of AA^T)
- Σ is an m×n diagonal matrix with non-negative entries called singular values
- V^T is the transpose of an n×n orthogonal matrix (columns of V are orthonormal eigenvectors of A^TA)
Think of this as the "ultimate" decomposition: unlike diagonalization (which only works for square matrices with enough eigenvectors), SVD works for every matrix—rectangular, singular, non-square, you name it.
Why It Matters
SVD reveals how a linear transformation stretches space along orthogonal directions:
- V^T rotates/reflects the input space
- Σ stretches along coordinate axes (by the singular values)
- U rotates/reflects into the output space
This is like saying every linear transformation is just a rotation, followed by pure stretching, followed by another rotation.
Connection to What You Know
You've seen the Spectral Theorem for symmetric matrices: A = QΛQ^T where Q is orthogonal and Λ diagonal. SVD generalizes this! When A is symmetric and positive semi-definite, SVD coincides with orthogonal diagonalization. But SVD doesn't require A to be square or symmetric.
The singular values in Σ are the square roots of the eigenvalues of A^TA (or AA^T).
Key Takeaway: SVD decomposes any matrix A into A = UΣV^T, where U and V are orthogonal matrices and Σ is diagonal—providing a complete geometric picture of the transformation regardless of the matrix's shape or properties.