Cross-Entropy Loss Function
You'll learn how logistic regression measures and minimizes prediction error using the log-loss (cross-entropy) cost function.
Why Not Mean Squared Error?
In linear regression, we used Mean Squared Error to measure how far predictions were from actual values. For classification, we need something different. Since logistic regression outputs probabilities (via the sigmoid function), we need a loss function that:
- Rewards confident correct predictions (predicting 0.99 when label is 1)
- Heavily penalizes confident wrong predictions (predicting 0.01 when label is 1)
- Works smoothly with probability outputs between 0 and 1
The Cross-Entropy Formula
For a single training example with true label y (0 or 1) and predicted probability ŷ:
Loss = -[y · log(ŷ) + (1 - y) · log(1 - ŷ)]
Why it works:
- If
y = 1: Loss =-log(ŷ). Asŷapproaches 1, loss approaches 0. Asŷapproaches 0, loss explodes to infinity. - If
y = 0: Loss =-log(1 - ŷ). Now we penalizeŷapproaching 1.
The logarithm creates an asymmetric penalty — being slightly wrong is forgiven, but being confidently wrong is catastrophic.
Analogy
Imagine grading a weather forecaster: if they say "90% chance of rain" and it rains, they get a small penalty. If they say "5% chance" and it pours, they get a massive penalty. Cross-entropy captures this asymmetry perfectly.
Cost Function Over Dataset
Average the loss across all m training examples:
J(θ) = -(1/m) Σ [y⁽ⁱ⁾ log(ŷ⁽ⁱ⁾) + (1 - y⁽ⁱ⁾) log(1 - ŷ⁽ⁱ⁾)]
This becomes our objective to minimize using gradient descent.
Key Takeaway: Cross-entropy (log-loss) is the standard loss function for binary classification, providing smooth gradients and severe penalties for confidently incorrect predictions.