Course contentsShow
← Machine Learning and Deep Learning
Lesson 484 of 3,53811. Model Evaluation MetricsFree lesson

Brier Score for Probabilistic Calibration

Measuring the mean squared difference between predicted probabilities and actual outcomes.

Brier Score for Probabilistic Calibration

What you'll learn: How to measure how well your model's predicted probabilities match actual outcomes using a single number.

What Is the Brier Score?

The Brier Score measures the mean squared difference between predicted probabilities and actual outcomes. Think of it as asking: "When my model says there's a 70% chance of rain, does it actually rain 70% of those times?"

Unlike metrics that only care if you got the label right (yes/no), Brier Score evaluates the quality of your probability estimates. A well-calibrated model doesn't just predict correctly—it expresses the right level of confidence.

How It Works

For each prediction, you calculate the squared difference between:

  • The predicted probability (e.g., 0.7 for "likely positive")
  • The actual outcome (1 if it happened, 0 if it didn't)

Then you average these squared differences across all predictions:

Brier Score = (1/N) × Σ(predicted_probability - actual_outcome)²

Understanding the Score

  • Range: 0 to 1 (lower is better)
  • Perfect score: 0.0 (predicted probabilities exactly match reality)
  • Worst score: 1.0 (maximally wrong probabilities)
  • Random baseline: ~0.25 for balanced binary problems

Example Intuition

If your model predicts 90% probability for positive class, but the actual outcome is 0 (negative), you get a large penalty: (0.9 - 0)² = 0.81. If it predicts 0.1 and actual is 0, the penalty is small: (0.1 - 0)² = 0.01.

Key Takeaway: The Brier Score quantifies probabilistic calibration by penalizing both incorrect predictions and overconfident/underconfident probability estimates—essential for applications where decision-making relies on probability values, not just class labels.