Course contentsShow
← Machine Learning and Deep Learning
Lesson 467 of 3,53811. Model Evaluation MetricsFree lesson

Brier Score for Probability Calibration

Measuring mean squared difference between predicted probabilities and actual outcomes.

Brier Score for Probability Calibration

What You'll Learn

You'll learn how to measure whether your classifier's predicted probabilities are well-calibrated using the Brier Score—a single number that tells you the quality of your probability predictions.

The Core Idea

The Brier Score measures how close your predicted probabilities are to the actual outcomes. It's the mean squared difference between what your model predicted (a probability between 0 and 1) and what actually happened (0 or 1).

Think of it like a weather forecaster saying "70% chance of rain." If it rains, you'd prefer they had said 100%. If it doesn't rain, you'd prefer they had said 0%. The Brier Score penalizes the squared distance from perfection.

How It Works

For each prediction:

  1. Your model outputs a probability (e.g., 0.8 for "positive class")
  2. The actual outcome is either 0 or 1
  3. Calculate the squared difference: (predicted - actual)²
  4. Average these squared differences across all predictions

Formula: Brier Score = (1/N) × Σ(predicted_probability - actual_outcome)²

Lower is better! A perfect score is 0 (every prediction exactly matched reality). The worst possible score is 1.

Interpretation

  • Brier Score = 0.20: Reasonably well-calibrated probabilities
  • Brier Score = 0.05: Excellent calibration
  • Brier Score > 0.25: Poor calibration—your probabilities may not reflect true likelihood

Unlike accuracy (which only cares about the final classification), Brier Score rewards confident, correct predictions and penalizes confident, incorrect ones. A model that always predicts 0.5 gets a Brier Score of 0.25—it's "maximally uncertain."

Key Takeaway: The Brier Score evaluates probability calibration by measuring the mean squared error between predicted probabilities and actual binary outcomes, with lower scores indicating better-calibrated predictions.