Least Squares Estimation in Multiple Regression
You'll learn how to derive the formulas that solve for all regression coefficients at once using matrix algebra.
The Core Idea
In simple linear regression, you found the best-fit line by minimizing squared errors. In multiple regression with several predictors, you're doing the same thing—but now you need to find multiple coefficients simultaneously. Matrix notation makes this elegant and efficient.
Recall the matrix formulation: Y = Xβ + ε, where:
- Y is your n×1 vector of responses
- X is your n×(p+1) design matrix (including a column of 1s for the intercept)
- β is your (p+1)×1 vector of unknown coefficients
- ε is your n×1 vector of errors
Deriving the Normal Equations
To minimize the sum of squared residuals, we want to minimize ||Y - Xβ||². Using calculus on this expression (taking the derivative with respect to β and setting it to zero), we arrive at the normal equations:
X'Xβ̂ = X'Y
Think of this as the multidimensional version of "setting the derivative to zero." The matrix X'X captures how your predictors relate to each other, while X'Y captures how they relate to the response.
Computing the Coefficient Estimates
If X'X is invertible (which requires your predictors to be linearly independent), we can solve for β̂:
β̂ = (X'X)⁻¹X'Y
This single formula gives you all coefficient estimates at once—intercept and slopes for every predictor. It's the direct generalization of the formulas you used in simple regression, but now handles any number of predictors.
Key Takeaway: The least squares estimates in multiple regression are found by solving X'Xβ̂ = X'Y, yielding β̂ = (X'X)⁻¹X'Y—a single matrix formula that computes all coefficients simultaneously.