Binary Cross-Entropy Loss
What you'll learn: How neural networks measure prediction quality for binary classification problems using log loss, and why it emerges naturally from probability theory.
The Core Idea
When your neural network predicts a probability for a binary outcome (yes/no, spam/ham, cat/dog), you need a loss function that heavily penalizes confident but wrong predictions. Binary Cross-Entropy Loss (also called log loss) does exactly this.
For a true label y (either 0 or 1) and predicted probability ŷ, the formula is:
Loss = -[y·log(ŷ) + (1-y)·log(1-ŷ)]
If the true class is 1, only the first term matters: -log(ŷ). If you predict ŷ=0.99 (confident and correct), the loss is tiny. If you predict ŷ=0.01 (confident but wrong), the loss explodes toward infinity.
Connection to Maximum Likelihood
This loss comes from maximum likelihood estimation. If your network outputs probability ŷ via a sigmoid activation (from logistic regression's foundations), you're modeling the likelihood that y=1. Maximizing the likelihood of seeing your training data is equivalent to minimizing the negative log-likelihood—which is exactly binary cross-entropy.
Think of it as: "How surprised should I be by the true label, given my predicted probabilities?" Lower surprise = better model.
Why Not Mean Squared Error?
MSE treats all errors equally. Binary cross-entropy grows exponentially as predictions become confidently wrong, creating stronger gradients that help the network learn faster and avoid getting stuck during training.
Key Takeaway: Binary cross-entropy is the principled loss function for binary classification, derived from maximizing the likelihood of observed data, and it naturally pairs with sigmoid outputs from logistic regression layers.