Course contentsShow
← Computer Science
Lesson 2830 of 2,87263. Classical ML AlgorithmsFree lesson

Cross-Entropy Loss Function

Learn the log-loss cost function that measures prediction error for classification problems.

Cross-Entropy Loss Function

You'll learn how logistic regression measures and minimizes prediction error using the log-loss (cross-entropy) cost function.

Why Not Mean Squared Error?

In linear regression, we used Mean Squared Error to measure how far predictions were from actual values. For classification, we need something different. Since logistic regression outputs probabilities (via the sigmoid function), we need a loss function that:

  1. Rewards confident correct predictions (predicting 0.99 when label is 1)
  2. Heavily penalizes confident wrong predictions (predicting 0.01 when label is 1)
  3. Works smoothly with probability outputs between 0 and 1

The Cross-Entropy Formula

For a single training example with true label y (0 or 1) and predicted probability ŷ:

Loss = -[y · log(ŷ) + (1 - y) · log(1 - ŷ)]

Why it works:

  • If y = 1: Loss = -log(ŷ). As ŷ approaches 1, loss approaches 0. As ŷ approaches 0, loss explodes to infinity.
  • If y = 0: Loss = -log(1 - ŷ). Now we penalize ŷ approaching 1.

The logarithm creates an asymmetric penalty — being slightly wrong is forgiven, but being confidently wrong is catastrophic.

Analogy

Imagine grading a weather forecaster: if they say "90% chance of rain" and it rains, they get a small penalty. If they say "5% chance" and it pours, they get a massive penalty. Cross-entropy captures this asymmetry perfectly.

Cost Function Over Dataset

Average the loss across all m training examples:

J(θ) = -(1/m) Σ [y⁽ⁱ⁾ log(ŷ⁽ⁱ⁾) + (1 - y⁽ⁱ⁾) log(1 - ŷ⁽ⁱ⁾)]

This becomes our objective to minimize using gradient descent.

Key Takeaway: Cross-entropy (log-loss) is the standard loss function for binary classification, providing smooth gradients and severe penalties for confidently incorrect predictions.