Course contentsShow
← Mathematics
Lesson 2238 of 3,12146. Linear Algebra — Inner Products and OrthogonalityFree lesson

Applications: Linear Regression

Apply least squares to fit linear models to data by minimizing the sum of squared residuals.

Applications: Linear Regression

What You'll Learn

You'll learn how to use the least squares method to find the best-fitting straight line through a scatter of data points.

The Core Problem

Imagine you've collected real-world data—say, hours studied vs. exam scores for 20 students. When you plot these points, they don't fall perfectly on a line. But you want to find the line that best represents the trend. How do you define "best"?

Linear regression answers this by finding the line that minimizes the total squared vertical distance from each data point to the line. This is exactly the least squares problem you've been studying!

From Data to Linear Model

Suppose you have data points (x1,y1),(x2,y2),…,(xm,ym)(x_1, y_1), (x_2, y_2), \ldots, (x_m, y_m). You want to fit a line y=mx+by = mx + b.

For each data point, there's a residual (error): the difference between the actual yiy_i and the predicted value mxi+bmx_i + b.

We want to minimize: Sum of squared residuals=∑i=1m(yi−(mxi+b))2\text{Sum of squared residuals} = \sum_{i=1}^{m} (y_i - (mx_i + b))^2

This is a least squares problem! In matrix form: Ax=bA\mathbf{x} = \mathbf{b}

where AA has rows [xi1][x_i \quad 1], x=[mb]\mathbf{x} = \begin{bmatrix} m \\ b \end{bmatrix}, and b=[y1y2⋮ym]\mathbf{b} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{bmatrix}.

Solving It

Use the normal equations: ATAx=ATbA^T A \mathbf{x} = A^T \mathbf{b}

Solving this gives you the optimal slope mm and intercept bb. The projection of b\mathbf{b} onto the column space of AA gives the "closest possible" predictions.

Why This Matters

Linear regression appears everywhere: predicting sales, analyzing trends, fitting scientific models. The least squares framework guarantees you get the line that minimizes prediction error in the squared-distance sense.

Key Takeaway: Linear regression applies least squares to find the best-fit line through data by minimizing the sum of squared residuals—transforming a practical curve-fitting problem into a solvable linear algebra equation.