Course contentsShow
← Machine Learning and Deep Learning
Lesson 20 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Orthogonality and Orthonormal Vectors

Understand orthogonal vectors, orthonormal bases, and their computational advantages in ML.

Orthogonality and Orthonormal Vectors

What You'll Learn

You'll discover what orthogonal and orthonormal vectors are, and why they make computations in ML dramatically simpler.

The Core Idea

Orthogonal vectors are vectors that meet at right angles (90 degrees). Mathematically, two vectors u and v are orthogonal if their dot product equals zero: u · v = 0.

Think of the x-axis and y-axis on a graph—they're perfectly perpendicular, never interfering with each other's direction. That's orthogonality.

Orthonormal vectors take this a step further: they're orthogonal and each has a norm (length) of exactly 1. An orthonormal basis is a set of orthonormal vectors that span a space—like a perfectly aligned, unit-length coordinate system.

Why This Matters in ML

When vectors in a basis are orthonormal, computations become incredibly efficient:

  1. Easy projections: Finding how much of one vector lies in the direction of another becomes a simple dot product (no division needed!)
  2. Stable numerics: Orthogonal matrices preserve lengths and angles, preventing numerical errors from accumulating
  3. Simplified inverses: The inverse of an orthogonal matrix is just its transpose—a trivial operation
  4. Decorrelated features: Orthogonal features don't redundantly encode the same information

This is why techniques like Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) produce orthonormal bases—they're leveraging these computational advantages.

Visualization

Imagine organizing a library: orthogonal categories (Fiction, Science, History) don't overlap. Orthonormal means each category is equally weighted and distinct—no book belongs to multiple shelves, making retrieval fast and unambiguous.

Key Takeaway: Orthogonal vectors are perpendicular (dot product = 0); orthonormal vectors are orthogonal with unit length. Using orthonormal bases simplifies ML computations and improves numerical stability.