Course contentsShow
← Machine Learning and Deep Learning
Lesson 3097 of 3,53868. Evaluation in PracticeFree lesson

Classification Task Evaluation Design

Selecting appropriate metrics for binary and multiclass classification based on class balance and cost asymmetry.

Classification Task Evaluation Design

What you'll learn: How to choose the right evaluation metrics for classification problems by considering class distribution and misclassification costs.

Why Accuracy Isn't Always Enough

When evaluating classification models in production, context matters more than convention. A 95% accurate fraud detector sounds impressive—until you realize fraud occurs in only 2% of transactions. A naive model predicting "no fraud" every time would achieve 98% accuracy but catch zero fraudsters.

Class Balance: The First Design Choice

Balanced classes (roughly equal positive/negative examples) allow straightforward metrics:

  • Accuracy works well when all errors hurt equally
  • Macro-averaged F1 treats each class fairly

Imbalanced classes demand different approaches:

  • Precision when false positives are costly (spam detection: don't lose real emails)
  • Recall when false negatives are costly (cancer screening: don't miss patients)
  • F-beta scores to weight precision/recall based on business priorities
  • AUC-ROC or AUC-PR for threshold-independent evaluation

Cost Asymmetry: The Business Lens

Not all mistakes cost the same. Missing a fraudulent 10,000transactionhurtsmorethanflaggingalegitimate10,000 transaction hurts more than flagging a legitimate 5 purchase for review. When costs differ:

  • Define a cost matrix assigning dollar values to each error type
  • Use expected cost as your optimization metric
  • Set decision thresholds based on cost ratios, not 0.5 defaults

Multiclass Considerations

For multiclass problems, choose between:

  • Micro-averaging (pool all predictions) when class sizes vary naturally
  • Macro-averaging (average per-class metrics) when all classes matter equally
  • Weighted averaging when business impact differs by class

Key Takeaway: Design your classification evaluation around class balance and business costs, not default metrics—the "best" metric depends on what mistakes your system can and cannot afford to make.