The Least Squares Problem
What You'll Learn
You'll learn how to formulate the problem of finding the "best" approximate solution when a linear system has no exact solution.
The Core Idea
Sometimes we need to solve a linear system Ax = b, but it's inconsistent—no exact solution exists. This happens frequently in real-world applications: you have more equations than unknowns (an overdetermined system), and the data contains measurement errors or noise.
Instead of giving up, we ask: What vector x̂ gets us as close as possible to b?
The Least Squares Problem is to find the vector x̂ that minimizes the distance between Ax and b. In other words, we want to minimize:
||Ax - b||²
This measures the squared length of the error (or residual) vector. Why squared? It's mathematically convenient and emphasizes larger errors.
The Geometric Insight
Think of it this way: b lives in ℝᵐ, but Ax can only reach vectors in the column space of A (call it Col(A)). If b isn't in Col(A), we can't solve Ax = b exactly.
The best we can do is find the vector Ax̂ in Col(A) that is closest to b. From your recent lesson on projections, you know this closest vector is the orthogonal projection of b onto Col(A)!
So the least squares problem becomes: find x̂ such that Ax̂ equals the projection of b onto Col(A).
Applications
Least squares is everywhere: fitting a line through scattered data points, signal processing, computer vision, machine learning, and more.
Key Takeaway: The least squares problem finds the best approximate solution x̂ to an inconsistent system Ax = b by minimizing ||Ax - b||², which geometrically means projecting b onto the column space of A.