DETR: Detection with Transformers
You'll learn how transformers revolutionized object detection by eliminating hand-crafted components like anchors and Non-Maximum Suppression.
The Breakthrough Idea
Until now, all the object detectors you've seen (YOLO, SSD, RetinaNet) relied on carefully engineered components: anchor boxes to propose regions and Non-Maximum Suppression (NMS) to filter duplicate predictions. DETR (DEtection TRansformer) throws these away entirely, treating object detection as a direct set prediction problem.
Think of traditional detectors as artists who sketch many rough drafts (anchors) then erase duplicates (NMS). DETR is like an artist who directly paints exactly the objects they see—no drafts, no erasing.
How DETR Works
DETR uses a transformer encoder-decoder architecture borrowed from natural language processing:
- CNN Backbone: Extracts image features (like ResNet-50)
- Transformer Encoder: Processes these spatial features with self-attention, learning relationships between different image regions
- Transformer Decoder: Takes learned queries (think of these as "slots" for objects) and predicts a fixed number of objects directly
- Set Prediction: Outputs exactly N predictions (e.g., 100), where each contains a class label and bounding box
The magic? DETR uses bipartite matching during training to assign each ground-truth object to exactly one prediction, eliminating the need for NMS. Empty slots predict "no object."
This end-to-end approach is elegant: you feed an image in, and you get detections out—no post-processing tricks needed.
Key Takeaway: DETR simplifies object detection by treating it as direct set prediction using transformers, eliminating hand-crafted components like anchors and NMS for a truly end-to-end learnable system.