Likelihood-Based Metrics Beyond Perplexity
What you'll learn: How to measure language model quality using calibrated log-likelihood and surprisal, going beyond simple perplexity.
Why Look Beyond Perplexity?
While perplexity tells you how "surprised" a model is on average, it doesn't capture everything about model quality. Alternative likelihood-based metrics give you more nuanced views of how well your language model understands and predicts text.
Calibrated Log-Likelihood
Calibrated log-likelihood adjusts raw probability estimates to account for model confidence. A well-calibrated model's predicted probabilities should match actual frequencies—if it says something will happen 70% of the time, it should indeed happen 70% of the time.
Think of it like a weather forecaster: perplexity measures whether they predict rain correctly, but calibration measures whether their "30% chance of rain" actually means it rains 3 out of 10 times.
When evaluating, you compare the model's assigned log-probabilities against observed outcomes across probability bins. Poor calibration means the model is overconfident or underconfident, even if average perplexity looks good.
Surprisal
Surprisal (also called information content) measures how unexpected a specific token is: surprisal = -log₂(p(token)). Unlike perplexity, which averages across all tokens, surprisal lets you analyze individual predictions.
High surprisal means "very surprising"—the model didn't expect this token. By analyzing surprisal distributions, you can identify which linguistic phenomena your model struggles with (rare words, long-distance dependencies, domain-specific terminology).
This is like examining not just your average test score, but which specific questions tripped you up.
Practical Value
These metrics help you understand why a model performs well or poorly on specific slices of data, complementing perplexity's aggregate view with granular insights about calibration quality and prediction patterns.
Key Takeaway: Calibrated log-likelihood reveals whether probability estimates are trustworthy, while surprisal provides token-level insight into prediction difficulty—both essential for understanding language model behavior beyond average perplexity.