Multi-Label Evaluation Metrics
What you'll learn: How to measure the success of multi-label classifiers using specialized metrics that account for partial correctness across multiple labels.
Why Standard Metrics Fall Short
In multi-label classification, an instance can have multiple correct labels simultaneously (like a photo tagged with "sunset," "beach," and "ocean"). Standard accuracy treats this as all-or-nothing: you're either completely right or completely wrong. But what if you predicted 2 out of 3 labels correctly? Standard metrics miss this nuance.
Instance-Based vs Label-Based Measures
Instance-based metrics evaluate predictions per example, then average across all instances. They answer: "How well did we do on each data point?"
Label-based metrics evaluate per label first, treating each label as a separate binary problem, then aggregate. They answer: "How well did we predict each individual label?"
Key Multi-Label Metrics
Hamming Loss: The fraction of labels incorrectly predicted (false positives + false negatives divided by total labels). Lower is better—it captures partial mistakes.
Subset Accuracy (Exact Match Ratio): The strictest metric—only counts predictions that match the true label set exactly. Often very low in practice.
F1 Scores with Different Averaging:
- Micro: Aggregate all label decisions, then compute F1 (treats all labels equally)
- Macro: Compute F1 per label, then average (treats rare labels same as common ones)
- Samples: Compute F1 per instance, then average (focuses on per-example performance)
Choosing the Right Metric
Use Hamming loss when partial credit matters. Use subset accuracy when only perfect predictions count. Choose micro-F1 for overall performance, macro-F1 when rare labels matter equally, and samples-F1 for instance-level quality.
Key Takeaway: Multi-label metrics capture partial correctness; Hamming loss and subset accuracy measure different extremes, while F1 variants (micro/macro/samples) offer nuanced views depending on whether you care more about labels, instances, or overall predictions.