Course contentsShow
← Data Science
Lesson 517 of 2,14514. Simple Linear RegressionFree lesson

The Least Squares Criterion

Explore why we minimize the sum of squared residuals rather than absolute errors or other criteria for fitting the line.

The Least Squares Criterion

You'll learn: Why we square residuals when finding the best-fit line, rather than simply adding up absolute distances.

When fitting a line to data, we need a mathematical rule to define "best fit." The least squares criterion says: choose the line that minimizes the sum of squared residuals.

Why Square the Residuals?

Remember that residuals are the vertical distances between observed data points and our predicted line. We could measure fit in several ways:

  • Sum of raw residuals? No—positive and negative errors cancel out. A terrible line could sum to zero!
  • Sum of absolute residuals? Better, but mathematically difficult to work with (no smooth derivative).
  • Sum of squared residuals? Perfect! ✓

Three Key Advantages of Squaring

  1. Eliminates sign problems: Squaring makes all errors positive, so they can't cancel each other out.

  2. Emphasizes larger errors: A point that's 4 units away contributes 16 to the sum, while one that's 2 units away contributes only 4. This heavily penalizes outliers and big mistakes, encouraging the line to fit the overall pattern well.

  3. Mathematical elegance: Squared terms have smooth derivatives, making it possible to solve for the optimal slope and intercept using calculus. We can find the exact answer, not just approximate it.

Think of it this way: If you're trying to balance a plank on a pivot, small wobbles are tolerable, but large tilts are disastrous. Squaring ensures we really care about avoiding big errors.

Key Takeaway: The least squares criterion minimizes the sum of squared residuals because squaring prevents cancellation, penalizes large errors appropriately, and yields a mathematically solvable optimization problem.