Singular Values and Their Meaning
What You'll Learn
You'll discover what singular values actually are mathematically and why they measure how much a matrix stretches space in different directions.
The Mathematical Definition
Singular values are the square roots of the eigenvalues of the matrix A^T A (or equivalently, A^H A for complex matrices). If λ₁, λ₂, ..., λₙ are the eigenvalues of A^T A, then the singular values are σᵢ = √λᵢ.
Since A^T A is always symmetric and positive semi-definite (recall from your earlier work with symmetric matrices), its eigenvalues are non-negative real numbers. This guarantees that taking square roots makes sense—no imaginary numbers appear.
What Do Singular Values Mean?
Singular values represent scaling factors: they tell you how much the matrix A stretches or compresses vectors in particular directions.
Think of it this way: when you apply a linear transformation A to a vector, it might:
- Stretch the vector by a factor of 5 in one direction (σ₁ = 5)
- Compress it by a factor of 0.5 in another direction (σ₂ = 0.5)
- Completely flatten it to zero in yet another direction (σ₃ = 0)
The largest singular value tells you the maximum possible stretching factor, while the smallest nonzero singular value tells you the minimum stretching in directions that aren't completely collapsed.
Connection to A^T A
Why A^T A? When you compute A^T A and find its eigenvalues, you're essentially measuring the "energy" or "variance" in each direction after the transformation. The square root brings these values back to the scale of the original transformation's stretching factors.
Key Takeaway: Singular values are the square roots of eigenvalues of A^T A and quantify how much a matrix scales space along its principal directions—they are the fundamental "stretching factors" of any linear transformation.