Course contentsShow
← Computer Science
Lesson 2832 of 2,87263. Classical ML AlgorithmsFree lesson

Maximum Likelihood Estimation

Understand logistic regression from a probabilistic perspective using maximum likelihood.

Maximum Likelihood Estimation

You'll learn: How logistic regression finds optimal parameters by maximizing the likelihood of observing your training data.

The Core Idea

Remember how we defined the cross-entropy loss for logistic regression? Maximum Likelihood Estimation (MLE) is the probabilistic principle that justifies why we use that particular loss function.

Here's the insight: Your logistic regression model outputs probabilities. For each training example, it predicts P(y=1|x). MLE asks: "What parameter values make our observed training labels most probable?"

From Probability to Parameters

Think of it like adjusting a coin's bias to best explain the outcomes you've seen. If you flipped 10 times and got 7 heads, MLE would estimate the coin is 70% biased toward heads—because that parameter maximizes the likelihood of seeing exactly what you observed.

For logistic regression:

  • Each prediction ŷᵢ = σ(wᵀxᵢ) is a probability
  • The likelihood of seeing label yᵢ is ŷᵢ if yᵢ=1, or (1-ŷᵢ) if yᵢ=0
  • The total likelihood across all training examples is the product of individual likelihoods

The Connection to Cross-Entropy

Maximizing likelihood is hard because products are unwieldy. Solution: take the log-likelihood (products become sums). Then maximize log-likelihood = minimize negative log-likelihood.

That negative log-likelihood? It's exactly the cross-entropy loss you've been using! So when you run gradient descent on cross-entropy, you're actually performing MLE—finding weights that make your training data most probable under the model.

Key Takeaway: Maximum Likelihood Estimation provides the probabilistic foundation for logistic regression's cross-entropy loss—you're finding parameters that maximize the probability of observing your training labels.