Maximum Likelihood Estimation in Logistic Regression
What you'll learn: Why logistic regression uses maximum likelihood instead of least squares to estimate coefficients.
Why Not Least Squares?
In linear regression, we minimize the squared differences between predicted and actual values. But logistic regression predicts probabilities of binary outcomes (0 or 1), not continuous values. Squaring errors between probabilities and binary outcomes doesn't make statistical sense—you can't minimize "squared distance" from a probability curve to discrete 0/1 points in a meaningful way.
The Maximum Likelihood Approach
Instead, logistic regression asks: "What coefficient values make our observed data most probable?"
Think of it like tuning a radio: you adjust the knobs (coefficients) until the signal (likelihood of observing your exact dataset) is strongest. Maximum likelihood estimation (MLE) finds the parameter values that maximize the probability of seeing the actual outcomes we observed.
How It Works
For each observation, the model predicts a probability p(y=1). Given your binary outcomes:
- If y=1 actually occurred, we want that probability to be high
- If y=0 actually occurred, we want that probability to be low (1-p)
The likelihood is the product of all these individual probabilities across your dataset. MLE finds coefficients that maximize this likelihood (or equivalently, maximize the log-likelihood, which is mathematically easier).
Unlike linear regression's closed-form solution, logistic regression requires iterative optimization algorithms (like gradient descent or Newton-Raphson) to find the maximum likelihood estimates numerically.
Why This Matters
MLE provides coefficients with desirable statistical properties: they're consistent, asymptotically normal, and efficient. This foundation allows all the inference tools you'll use—standard errors, confidence intervals, and hypothesis tests.
Key Takeaway: Logistic regression uses maximum likelihood estimation because it finds coefficients that make your observed binary outcomes most probable, unlike least squares which only works well for continuous outcomes.