Course contentsShow
← Mathematics
Lesson 2236 of 3,12146. Linear Algebra — Inner Products and OrthogonalityFree lesson

Least Squares via Orthogonal Projection

Interpret the least squares solution as the orthogonal projection of b onto the column space of A.

Least Squares via Orthogonal Projection

You'll learn how the least squares solution is actually the orthogonal projection of your target vector onto the column space of your matrix.

The Geometric Heart of Least Squares

When solving Ax = b has no exact solution (the system is inconsistent), we look for the "best approximate solution" by minimizing ||Ax - b||². Here's the beautiful insight: the least squares solution geometrically means finding the point in the column space of A that's closest to b.

Think of it like this: Imagine b is a point floating somewhere in space, and the column space of A is a plane (or subspace) below it. The best approximation Ax̂ is the shadow that b casts when you shine a light straight down onto that plane—that's the orthogonal projection of b onto Col(A).

Why This Works

The projection proj_Col(A)(b) minimizes the distance ||b - Ax|| because the shortest path from a point to a subspace is always perpendicular to that subspace. The error vector e = b - Ax̂ points straight out, orthogonal to everything in Col(A).

This is exactly what the normal equations A^T Ax̂ = A^T b guarantee: they enforce that A^T(b - Ax̂) = 0, meaning the error is perpendicular to every column of A—in other words, orthogonal to the column space.

The Connection

  • Least squares solution x̂ produces Ax̂
  • Ax̂ is the orthogonal projection of b onto Col(A)
  • Minimizing the error = finding the projection

Key Takeaway: The least squares solution isn't just algebraic manipulation—it's the natural geometric answer: project your target vector onto the space you can reach, and work backward to find which input gives you that projection.