Deriving the Least Squares Estimators
You'll learn: How calculus proves that the slope and intercept formulas minimize the sum of squared residuals.
The Mathematical Foundation
You already know the least squares criterion—we want to minimize the sum of squared residuals (SSR). Now we'll see why the formulas for slope and intercept work.
For a line ŷ = b₀ + b₁x, each residual is:
- eᵢ = yᵢ - (b₀ + b₁xᵢ)
The sum of squared residuals becomes:
- SSR = Σ[yᵢ - (b₀ + b₁xᵢ)]²
Using Calculus to Find the Minimum
Think of SSR as a bowl-shaped surface. The minimum point sits at the bottom. Calculus finds this by taking partial derivatives with respect to b₀ and b₁, then setting them equal to zero.
For the slope (b₁): Taking the derivative and solving yields:
- b₁ = Σ[(xᵢ - x̄)(yᵢ - ȳ)] / Σ[(xᵢ - x̄)²]
This is the covariance of x and y divided by the variance of x.
For the intercept (b₀): The derivative gives us:
- b₀ = ȳ - b₁x̄
This ensures the line passes through the point (x̄, ȳ)—the center of your data cloud.
Why This Matters
These aren't arbitrary formulas someone guessed—they're mathematically proven to give the unique line that minimizes squared prediction errors. Any other slope or intercept would produce a larger SSR.
Key Takeaway: The least squares estimators for slope and intercept are derived using calculus by minimizing the sum of squared residuals, producing formulas that guarantee the best-fitting line.