Orthogonality and Orthonormal Vectors
What You'll Learn
You'll discover what orthogonal and orthonormal vectors are, and why they make computations in ML dramatically simpler.
The Core Idea
Orthogonal vectors are vectors that meet at right angles (90 degrees). Mathematically, two vectors u and v are orthogonal if their dot product equals zero: u · v = 0.
Think of the x-axis and y-axis on a graph—they're perfectly perpendicular, never interfering with each other's direction. That's orthogonality.
Orthonormal vectors take this a step further: they're orthogonal and each has a norm (length) of exactly 1. An orthonormal basis is a set of orthonormal vectors that span a space—like a perfectly aligned, unit-length coordinate system.
Why This Matters in ML
When vectors in a basis are orthonormal, computations become incredibly efficient:
- Easy projections: Finding how much of one vector lies in the direction of another becomes a simple dot product (no division needed!)
- Stable numerics: Orthogonal matrices preserve lengths and angles, preventing numerical errors from accumulating
- Simplified inverses: The inverse of an orthogonal matrix is just its transpose—a trivial operation
- Decorrelated features: Orthogonal features don't redundantly encode the same information
This is why techniques like Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) produce orthonormal bases—they're leveraging these computational advantages.
Visualization
Imagine organizing a library: orthogonal categories (Fiction, Science, History) don't overlap. Orthonormal means each category is equally weighted and distinct—no book belongs to multiple shelves, making retrieval fast and unambiguous.
Key Takeaway: Orthogonal vectors are perpendicular (dot product = 0); orthonormal vectors are orthogonal with unit length. Using orthonormal bases simplifies ML computations and improves numerical stability.