Course contentsShow
← Machine Learning and Deep Learning
Lesson 547 of 3,53813. Imbalanced and Multi-Label ProblemsFree lesson

Focal Loss and Hard Example Mining

Derive focal loss for down-weighting easy examples, focusing gradient updates on hard misclassified samples in imbalanced sets.

Focal Loss and Hard Example Mining

What You'll Learn

You'll learn how Focal Loss reshapes the standard loss function to automatically focus training on hard, misclassified examples while reducing the influence of easy, correctly classified ones—especially powerful for imbalanced datasets.

The Problem with Easy Examples

In imbalanced datasets, your model might correctly classify thousands of easy majority-class examples but struggle with a few critical minority-class samples. Standard cross-entropy loss treats all examples equally, so the overwhelming number of easy examples dominates the gradient updates, leaving hard cases undertrained.

How Focal Loss Works

Focal Loss modifies cross-entropy by adding a modulating factor that down-weighs well-classified examples:

Focal Loss = -(1 - p)^γ * log(p)

Where:

  • p is the predicted probability for the true class
  • γ (gamma) is a focusing parameter (typically 2)

The magic: When an example is correctly classified (p close to 1), (1-p)^γ becomes tiny, reducing its loss contribution. When misclassified (p near 0), (1-p)^γ stays large, keeping the loss high.

Think of it like a teacher spending more time with struggling students rather than those who already understand—the gradient "budget" goes where it's needed most.

Hard Example Mining

This is the broader strategy: identify difficult examples dynamically during training and prioritize them. Focal Loss does this automatically through its formula, but you can also explicitly mine hard negatives (challenging false positives) in tasks like object detection.

Key Takeaway: Focal Loss automatically emphasizes hard, misclassified examples by reducing the loss contribution of easy, well-classified ones—making your model learn from its mistakes rather than being distracted by what it already knows.