Course contentsShow
← Machine Learning and Deep Learning
Lesson 192 of 3,5383. Linear RegressionFree lesson

The Least Squares Criterion

Understanding the goal of minimizing sum of squared errors to find the best-fitting line.

The Least Squares Criterion

You'll learn: Why "least squares" is the standard method for finding the best-fitting line in linear regression.

The Core Idea

You already know that residuals measure how far each prediction misses the actual value, and that Mean Squared Error (MSE) calculates the average of squared residuals. The Least Squares Criterion is simply the principle that the best line is the one that minimizes the sum of squared errors.

Think of it like this: imagine you're trying to place a plank of wood so it's as close as possible to a bunch of scattered nails sticking up from the floor. "Least squares" says the best position minimizes the total of all squared vertical distances from each nail to the plank.

Why Squared Errors?

Why not just minimize the sum of errors directly? Because positive and negative errors would cancel out—a terrible prediction could look perfect! Why not absolute errors? We could, but squared errors have beautiful mathematical properties:

  • Squaring penalizes large errors more heavily (a residual of 10 contributes 100, while five residuals of 2 each contribute only 20 total)
  • The resulting optimization problem has a clean, exact solution
  • The derivatives are smooth and well-behaved for optimization algorithms

The Connection to What You Know

This criterion is exactly why we use MSE as our loss function. When you fit a linear regression model with scikit-learn, it's finding the parameters (slope and intercept) that satisfy this least squares criterion—the values that make the sum of squared residuals as small as mathematically possible.

Key Takeaway: The Least Squares Criterion defines "best fit" as the line that minimizes the sum of squared residuals, giving us a clear mathematical objective for linear regression.