Brier Score for Probability Calibration
What You'll Learn
You'll learn how to measure whether your classifier's predicted probabilities are well-calibrated using the Brier Score—a single number that tells you the quality of your probability predictions.
The Core Idea
The Brier Score measures how close your predicted probabilities are to the actual outcomes. It's the mean squared difference between what your model predicted (a probability between 0 and 1) and what actually happened (0 or 1).
Think of it like a weather forecaster saying "70% chance of rain." If it rains, you'd prefer they had said 100%. If it doesn't rain, you'd prefer they had said 0%. The Brier Score penalizes the squared distance from perfection.
How It Works
For each prediction:
- Your model outputs a probability (e.g., 0.8 for "positive class")
- The actual outcome is either 0 or 1
- Calculate the squared difference: (predicted - actual)²
- Average these squared differences across all predictions
Formula: Brier Score = (1/N) × Σ(predicted_probability - actual_outcome)²
Lower is better! A perfect score is 0 (every prediction exactly matched reality). The worst possible score is 1.
Interpretation
- Brier Score = 0.20: Reasonably well-calibrated probabilities
- Brier Score = 0.05: Excellent calibration
- Brier Score > 0.25: Poor calibration—your probabilities may not reflect true likelihood
Unlike accuracy (which only cares about the final classification), Brier Score rewards confident, correct predictions and penalizes confident, incorrect ones. A model that always predicts 0.5 gets a Brier Score of 0.25—it's "maximally uncertain."
Key Takeaway: The Brier Score evaluates probability calibration by measuring the mean squared error between predicted probabilities and actual binary outcomes, with lower scores indicating better-calibrated predictions.