Choosing Metrics Based on Cost Functions
What you'll learn: How to select the right evaluation metric by understanding the real-world costs of different types of errors.
The Business Cost Perspective
Not all mistakes cost the same. In machine learning, asymmetric costs mean that false positives and false negatives carry different real-world consequences. Choosing your evaluation metric should reflect these actual business or operational costs.
Think of medical screening: A false negative (missing a disease) might cost a life, while a false positive (flagging a healthy patient) costs time and money for follow-up tests. These aren't equal losses—one is clearly more expensive than the other.
Mapping Costs to Metrics
When false negatives are more expensive (like disease detection, fraud detection where you miss real fraud), prioritize Recall or use F-Beta Score with β > 1 to weight recall more heavily.
When false positives are more expensive (like spam filters blocking important emails, or wrongly accusing customers of fraud), prioritize Precision or use F-Beta Score with β < 1.
When costs are roughly equal, F1 Score or Accuracy (if classes are balanced) work well.
For quantifiable monetary costs, you might create a custom cost function: Total Cost = (FP × cost_fp) + (FN × cost_fn). Some organizations optimize directly on expected cost rather than traditional metrics.
Example Scenario
Imagine a loan default predictor where:
- Missing a default (false negative) costs $10,000 in lost money
- Rejecting a good applicant (false positive) costs $500 in lost business
Your metric choice should heavily penalize false negatives—Recall becomes your north star, possibly accepting lower Precision.
Key Takeaway: Always select evaluation metrics that align with the asymmetric business costs of your false positives and false negatives, not just what's mathematically convenient.