Course contentsShow
← Machine Learning and Deep Learning
Lesson 616 of 3,53815. Neural Network FoundationsFree lesson

Binary Cross-Entropy Loss

Understanding log loss for binary classification, deriving it from maximum likelihood, and its connection to logistic regression.

Binary Cross-Entropy Loss

What you'll learn: How neural networks measure prediction quality for binary classification problems using log loss, and why it emerges naturally from probability theory.

The Core Idea

When your neural network predicts a probability for a binary outcome (yes/no, spam/ham, cat/dog), you need a loss function that heavily penalizes confident but wrong predictions. Binary Cross-Entropy Loss (also called log loss) does exactly this.

For a true label y (either 0 or 1) and predicted probability ŷ, the formula is:

Loss = -[y·log(ŷ) + (1-y)·log(1-ŷ)]

If the true class is 1, only the first term matters: -log(ŷ). If you predict ŷ=0.99 (confident and correct), the loss is tiny. If you predict ŷ=0.01 (confident but wrong), the loss explodes toward infinity.

Connection to Maximum Likelihood

This loss comes from maximum likelihood estimation. If your network outputs probability ŷ via a sigmoid activation (from logistic regression's foundations), you're modeling the likelihood that y=1. Maximizing the likelihood of seeing your training data is equivalent to minimizing the negative log-likelihood—which is exactly binary cross-entropy.

Think of it as: "How surprised should I be by the true label, given my predicted probabilities?" Lower surprise = better model.

Why Not Mean Squared Error?

MSE treats all errors equally. Binary cross-entropy grows exponentially as predictions become confidently wrong, creating stronger gradients that help the network learn faster and avoid getting stuck during training.

Key Takeaway: Binary cross-entropy is the principled loss function for binary classification, derived from maximizing the likelihood of observed data, and it naturally pairs with sigmoid outputs from logistic regression layers.