Classification Task Evaluation Design
What you'll learn: How to choose the right evaluation metrics for classification problems by considering class distribution and misclassification costs.
Why Accuracy Isn't Always Enough
When evaluating classification models in production, context matters more than convention. A 95% accurate fraud detector sounds impressive—until you realize fraud occurs in only 2% of transactions. A naive model predicting "no fraud" every time would achieve 98% accuracy but catch zero fraudsters.
Class Balance: The First Design Choice
Balanced classes (roughly equal positive/negative examples) allow straightforward metrics:
- Accuracy works well when all errors hurt equally
- Macro-averaged F1 treats each class fairly
Imbalanced classes demand different approaches:
- Precision when false positives are costly (spam detection: don't lose real emails)
- Recall when false negatives are costly (cancer screening: don't miss patients)
- F-beta scores to weight precision/recall based on business priorities
- AUC-ROC or AUC-PR for threshold-independent evaluation
Cost Asymmetry: The Business Lens
Not all mistakes cost the same. Missing a fraudulent 5 purchase for review. When costs differ:
- Define a cost matrix assigning dollar values to each error type
- Use expected cost as your optimization metric
- Set decision thresholds based on cost ratios, not 0.5 defaults
Multiclass Considerations
For multiclass problems, choose between:
- Micro-averaging (pool all predictions) when class sizes vary naturally
- Macro-averaging (average per-class metrics) when all classes matter equally
- Weighted averaging when business impact differs by class
Key Takeaway: Design your classification evaluation around class balance and business costs, not default metrics—the "best" metric depends on what mistakes your system can and cannot afford to make.