Maximum Likelihood Estimation
You'll learn: How logistic regression finds optimal parameters by maximizing the likelihood of observing your training data.
The Core Idea
Remember how we defined the cross-entropy loss for logistic regression? Maximum Likelihood Estimation (MLE) is the probabilistic principle that justifies why we use that particular loss function.
Here's the insight: Your logistic regression model outputs probabilities. For each training example, it predicts P(y=1|x). MLE asks: "What parameter values make our observed training labels most probable?"
From Probability to Parameters
Think of it like adjusting a coin's bias to best explain the outcomes you've seen. If you flipped 10 times and got 7 heads, MLE would estimate the coin is 70% biased toward heads—because that parameter maximizes the likelihood of seeing exactly what you observed.
For logistic regression:
- Each prediction ŷᵢ = σ(wᵀxᵢ) is a probability
- The likelihood of seeing label yᵢ is ŷᵢ if yᵢ=1, or (1-ŷᵢ) if yᵢ=0
- The total likelihood across all training examples is the product of individual likelihoods
The Connection to Cross-Entropy
Maximizing likelihood is hard because products are unwieldy. Solution: take the log-likelihood (products become sums). Then maximize log-likelihood = minimize negative log-likelihood.
That negative log-likelihood? It's exactly the cross-entropy loss you've been using! So when you run gradient descent on cross-entropy, you're actually performing MLE—finding weights that make your training data most probable under the model.
Key Takeaway: Maximum Likelihood Estimation provides the probabilistic foundation for logistic regression's cross-entropy loss—you're finding parameters that maximize the probability of observing your training labels.