Course contentsShow
← Mathematics
Lesson 3062 of 3,12160. Statistics — ModelingFree lesson

The Least Squares Criterion

Learn how the best-fit line minimizes the sum of squared vertical distances from points to the line.

The Least Squares Criterion

You'll learn: How the best-fit regression line is chosen by minimizing the sum of squared vertical distances from data points to the line.

The Core Idea

When fitting a line to data in simple linear regression, we need a principled way to decide which line is "best." The least squares criterion provides this: among all possible lines, we choose the one that makes the sum of squared vertical distances (residuals) from each data point to the line as small as possible.

Think of it like this: imagine you're hanging a picture frame on a wall with several crooked nails. You want to position the frame so the total "badness" of how far each nail sticks out is minimized. Instead of just adding distances (which could cancel out), we square each distance first—this penalizes large errors more heavily and ensures all deviations count positively.

Why Squared Distances?

We square the vertical distances for three important reasons:

  1. Sign doesn't matter: Points above and below the line both contribute positively
  2. Large errors hurt more: A point far from the line gets heavily penalized (distance of 2 becomes 4, distance of 3 becomes 9)
  3. Mathematical tractability: Squared terms have nice calculus properties that let us find the minimum algebraically

The vertical distance from a point (x, y) to our proposed line ŷ = a + bx is simply |y - ŷ|. We square this, getting (y - ŷ)², and sum over all n data points: Σ(yᵢ - ŷᵢ)². The line that minimizes this sum is our least squares regression line.

Key Takeaway: The least squares criterion selects the regression line by minimizing Σ(yᵢ - ŷᵢ)², ensuring the best overall fit by making squared vertical deviations from points to the line as small as possible.