Choosing Performance Metrics for Importance
What You'll Learn
You'll learn how to select the right performance metric when measuring feature importance through permutation, ensuring your importance scores truly reflect what matters for your model's task.
The Core Idea
When you permute a feature to measure its importance, you need to quantify how much worse your model performs without that feature. But "worse" means different things for different problems—and choosing the wrong metric can completely mislead your interpretation.
The metric you choose for permutation importance should mirror the metric you care about in production. If you're optimizing for F1 score in your fraud detection model, use F1 score to measure performance degradation when features are shuffled. If you're predicting house prices and care about RMSE, measure importance using RMSE changes.
Why This Matters
Imagine two features in a cancer diagnosis model:
- Feature A: slightly improves accuracy from 95% to 96%
- Feature B: improves recall of rare positive cases from 60% to 85%
If you measure importance using accuracy, Feature A might appear more important. But if you measure using F1 score or recall, Feature B's life-saving contribution becomes clear. Different metrics highlight different aspects of performance.
Common Metric Choices
- Classification: accuracy (balanced classes), F1 score (imbalanced), AUC-ROC (ranking quality)
- Regression: RMSE (penalizes large errors), MAE (robust to outliers), R² (proportion of variance explained)
- Custom metrics: Use whatever your business actually cares about—conversion rate, revenue impact, fairness metrics
Key Takeaway: Always measure permutation importance using the same performance metric that drives your real-world decision-making; misaligned metrics produce importance scores that don't reflect what actually matters for your application.