Maximum Likelihood Estimation for Classification
What you'll learn: How to derive the likelihood function for binary classification by treating each prediction as a Bernoulli trial.
The Core Idea
In logistic regression, we need a principled way to find the best parameters (weights and bias). Maximum Likelihood Estimation (MLE) gives us exactly that by asking: "What parameter values make our observed training data most probable?"
Since each prediction in binary classification is either 0 or 1, we can model it as a Bernoulli distribution—like a weighted coin flip where our sigmoid function gives the probability of "heads" (class 1).
From Bernoulli to Likelihood
For a single training example with true label y and predicted probability p:
- If y = 1: we want p to be high
- If y = 0: we want (1 - p) to be high
The Bernoulli probability captures both cases elegantly:
P(y | p) = p^y × (1 - p)^(1-y)
For the entire dataset, assuming independence, we multiply these probabilities together. This product is called the likelihood function:
L(θ) = ∏ p_i^y_i × (1 - p_i)^(1-y_i)
where θ represents our model parameters, and each p_i comes from applying the sigmoid function to our linear combination of features.
Why This Matters
Maximizing this likelihood means finding parameters that make the training labels we actually observed as probable as possible. In practice, we maximize the log-likelihood (turning products into sums) to make computation easier and more stable.
Key Takeaway: Maximum Likelihood Estimation frames logistic regression as finding parameters that maximize the probability of observing our training labels, treating each prediction as a Bernoulli random variable.