DETR Loss Functions and Training
What you'll learn: How DETR combines multiple loss functions with bipartite matching to train an end-to-end object detector without hand-crafted components.
The Core Idea
DETR needs to predict both what objects are present (classification) and where they are (bounding boxes). Because DETR makes a fixed set of predictions (e.g., 100 object queries) but images contain varying numbers of objects, the training loss must handle this mismatch intelligently.
The solution: First use the Hungarian algorithm (bipartite matching) to pair each ground-truth object with exactly one predicted query. Then compute losses only on these optimal pairs.
The Three Loss Components
1. Class Loss
A cross-entropy loss on the predicted class probabilities. For unmatched queries (predicting "no object"), this encourages the model to predict the background class.
2. Bounding Box Loss
An L1 loss measuring the direct distance between predicted and ground-truth box coordinates. This is scale-sensitive—a 10-pixel error matters more for small objects than large ones.
3. GIoU Loss (Generalized Intersection over Union)
Measures how well boxes overlap, scale-invariant and better than L1 alone. GIoU penalizes both poor overlap and boxes that are far apart, even when they don't intersect at all.
The final loss is a weighted sum: Loss = λ_class * class_loss + λ_bbox * bbox_loss + λ_giou * giou_loss
Training Flow
- Forward pass produces 100 predictions
- Hungarian matching pairs predictions to ground-truth
- Compute losses only on matched pairs
- Backpropagate to update both transformer and learned object queries
Key Takeaway: DETR training combines classification, bounding box, and GIoU losses with bipartite matching to enable true end-to-end learning without anchor boxes or NMS post-processing.