Course contentsShow
← Machine Learning and Deep Learning
Lesson 554 of 3,53813. Imbalanced and Multi-Label ProblemsFree lesson

Multi-Label Evaluation Metrics

Understanding metrics like Hamming loss, subset accuracy, F1 micro/macro/samples, and label-based vs instance-based measures.

Multi-Label Evaluation Metrics

What you'll learn: How to measure the success of multi-label classifiers using specialized metrics that account for partial correctness across multiple labels.

Why Standard Metrics Fall Short

In multi-label classification, an instance can have multiple correct labels simultaneously (like a photo tagged with "sunset," "beach," and "ocean"). Standard accuracy treats this as all-or-nothing: you're either completely right or completely wrong. But what if you predicted 2 out of 3 labels correctly? Standard metrics miss this nuance.

Instance-Based vs Label-Based Measures

Instance-based metrics evaluate predictions per example, then average across all instances. They answer: "How well did we do on each data point?"

Label-based metrics evaluate per label first, treating each label as a separate binary problem, then aggregate. They answer: "How well did we predict each individual label?"

Key Multi-Label Metrics

Hamming Loss: The fraction of labels incorrectly predicted (false positives + false negatives divided by total labels). Lower is better—it captures partial mistakes.

Subset Accuracy (Exact Match Ratio): The strictest metric—only counts predictions that match the true label set exactly. Often very low in practice.

F1 Scores with Different Averaging:

  • Micro: Aggregate all label decisions, then compute F1 (treats all labels equally)
  • Macro: Compute F1 per label, then average (treats rare labels same as common ones)
  • Samples: Compute F1 per instance, then average (focuses on per-example performance)

Choosing the Right Metric

Use Hamming loss when partial credit matters. Use subset accuracy when only perfect predictions count. Choose micro-F1 for overall performance, macro-F1 when rare labels matter equally, and samples-F1 for instance-level quality.

Key Takeaway: Multi-label metrics capture partial correctness; Hamming loss and subset accuracy measure different extremes, while F1 variants (micro/macro/samples) offer nuanced views depending on whether you care more about labels, instances, or overall predictions.