Likelihood Function for Regression
What you'll learn: How to express the probability of observing your training data given specific model parameters, forming the foundation for maximum likelihood estimation in regression.
The Core Idea
In Bayesian and probabilistic machine learning, we don't just want point predictions—we want to understand the probability of our observed data. The likelihood function answers: "Given my model parameters (like weights and bias), how probable is the data I actually observed?"
For regression, this connects directly to an assumption you've been using all along: that predictions have Gaussian noise.
From Predictions to Probabilities
In linear regression, we typically write:
ŷ = wx + b
But probabilistically, we assume:
y = wx + b + ε, where ε ~ N(0, σ²)
This means each true target value y comes from a normal distribution centered at our prediction, with some variance σ².
The Likelihood Function
The likelihood is the probability of all observed targets {y₁, y₂, ..., yₙ} given parameters θ (weights, bias) and inputs X:
L(θ | X, y) = P(y | X, θ)
For independent observations with Gaussian noise:
L(θ) = ∏ᵢ (1/√(2πσ²)) exp(-(yᵢ - ŷᵢ)²/(2σ²))
Each data point contributes a Gaussian probability density centered at the model's prediction.
Why It Matters
Maximizing the likelihood leads directly to minimizing squared error—the loss function you already know! Taking the log (log-likelihood) turns products into sums and reveals that maximizing likelihood = minimizing MSE.
This probabilistic view opens the door to uncertainty quantification and Bayesian methods.
Key Takeaway: The likelihood function expresses how probable your observed data is under your model's parameters; under Gaussian noise assumptions, maximizing likelihood in regression is equivalent to minimizing mean squared error.