Course contentsShow
← Machine Learning and Deep Learning
Lesson 1367 of 3,53830. Vision TransformersFree lesson

DETR Loss Functions and Training

Class loss, bounding box loss, and GIoU loss combined with matching for end-to-end training.

DETR Loss Functions and Training

What you'll learn: How DETR combines multiple loss functions with bipartite matching to train an end-to-end object detector without hand-crafted components.

The Core Idea

DETR needs to predict both what objects are present (classification) and where they are (bounding boxes). Because DETR makes a fixed set of predictions (e.g., 100 object queries) but images contain varying numbers of objects, the training loss must handle this mismatch intelligently.

The solution: First use the Hungarian algorithm (bipartite matching) to pair each ground-truth object with exactly one predicted query. Then compute losses only on these optimal pairs.

The Three Loss Components

1. Class Loss

A cross-entropy loss on the predicted class probabilities. For unmatched queries (predicting "no object"), this encourages the model to predict the background class.

2. Bounding Box Loss

An L1 loss measuring the direct distance between predicted and ground-truth box coordinates. This is scale-sensitive—a 10-pixel error matters more for small objects than large ones.

3. GIoU Loss (Generalized Intersection over Union)

Measures how well boxes overlap, scale-invariant and better than L1 alone. GIoU penalizes both poor overlap and boxes that are far apart, even when they don't intersect at all.

The final loss is a weighted sum: Loss = λ_class * class_loss + λ_bbox * bbox_loss + λ_giou * giou_loss

Training Flow

  1. Forward pass produces 100 predictions
  2. Hungarian matching pairs predictions to ground-truth
  3. Compute losses only on matched pairs
  4. Backpropagate to update both transformer and learned object queries

Key Takeaway: DETR training combines classification, bounding box, and GIoU losses with bipartite matching to enable true end-to-end learning without anchor boxes or NMS post-processing.