Normal Equations
You'll learn how to derive and solve the system A^T Ax = A^T b to find the best approximate solution when Ax = b has no exact solution.
Why Normal Equations?
When you're faced with an inconsistent system Ax = b (more equations than unknowns, or incompatible constraints), you can't solve it exactly. But you can find the vector x that minimizes the error ||b - Ax||². This is the least squares solution, and the normal equations give you a direct way to compute it.
Deriving the Normal Equations
Remember from the last lesson: the least squares solution occurs when the error vector (b - Ax) is orthogonal to the column space of A. Mathematically, this means:
A^T(b - Ax) = 0
Expanding this:
- A^T b - A^T Ax = 0
- A^T Ax = A^T b
These are the normal equations. The name comes from "normal" meaning perpendicular—the residual is normal to the column space.
Solving the Normal Equations
If A has full column rank (linearly independent columns), then A^T A is invertible, and you can solve:
x = (A^T A)^(-1) A^T b
This gives you the unique least squares solution. Notice how the projection matrix P = A(A^T A)^(-1)A^T appears here—you're essentially projecting b onto the column space of A.
Example Structure
For a simple case where A is 3×2 (three data points, fitting a line), A^T A becomes a 2×2 matrix you can invert by hand, and A^T b is a 2×1 vector. Solve the 2×2 system to get your best-fit parameters.
Key Takeaway: The normal equations A^T Ax = A^T b transform an unsolvable inconsistent system into a solvable one that minimizes squared error, leveraging orthogonality to find the closest possible solution.