Binary Cross-Entropy Loss
What you'll learn: How the log-loss function measures the cost of incorrect probabilistic predictions in logistic regression.
The Core Idea
In logistic regression, your model doesn't just predict "yes" or "no"—it predicts the probability that something belongs to a class (like "80% chance this email is spam"). But how do you measure how wrong a probability prediction is?
Binary Cross-Entropy Loss (also called log-loss) is the cost function that penalizes confident wrong predictions heavily while gently correcting uncertain ones. It's the negative log-likelihood we use to train logistic regression models.
Why Negative Log-Likelihood?
Remember that Maximum Likelihood Estimation tries to maximize the probability of seeing your training data. Since optimization algorithms typically minimize cost functions, we take the negative of the log-likelihood—turning our maximization problem into a minimization one.
For a single prediction:
- If the true label is 1: Loss = -log(predicted_probability)
- If the true label is 0: Loss = -log(1 - predicted_probability)
The Penalty Structure
Think of it like weather forecasting: if you predict "90% chance of rain" and it rains, you're rewarded (low loss). But if you predict "90% chance of rain" and it's sunny, you pay a heavy penalty (high loss). Uncertain predictions like "50%" result in moderate loss regardless of outcome.
The logarithm creates this asymmetric penalty: being confidently wrong hurts exponentially more than being slightly off.
Key Takeaway: Binary cross-entropy loss is the negative log-likelihood function that measures how well your logistic regression's probability predictions match the actual labels, penalizing confident mistakes more severely than uncertain ones.