DETR: Detection Transformer Architecture
What You'll Learn
You'll discover how DETR revolutionizes object detection by treating it as a direct set prediction problem using Transformers, eliminating the need for hand-crafted anchors and post-processing.
The Problem with Traditional Detection
Previous object detection methods (like Faster R-CNN, YOLO) relied on anchors—predefined boxes at various scales and aspect ratios—and Non-Maximum Suppression (NMS) to filter duplicate predictions. These were engineering tricks, not learned components.
DETR's Revolutionary Approach
DETR (DEtection TRansformer) treats object detection as a set prediction problem. Instead of predicting thousands of overlapping boxes and filtering them, DETR directly predicts a fixed-size set of objects in one shot.
Think of it like this: traditional detectors are like casting a wide net with many holes, then manually removing duplicates. DETR is like having a smart fishing system that picks out exactly N fish (objects) in parallel, with no duplicates by design.
Architecture Components
- CNN Backbone: Extracts image features (like ResNet)
- Transformer Encoder: Processes flattened spatial features with self-attention
- Transformer Decoder: Takes learned object queries (learnable embeddings) and attends to encoder features
- Prediction Heads: Each decoder output predicts one object (class + bounding box)
The Magic: Bipartite Matching
DETR predicts a fixed set of N predictions (say, 100). During training, it uses Hungarian algorithm to match predictions to ground-truth objects optimally. Unmatched predictions learn to output "no object."
No More Anchors or NMS
Because each object query specializes in detecting certain object types/locations through training, predictions naturally don't overlap—eliminating NMS entirely!
Key Takeaway: DETR simplifies object detection by framing it as direct set prediction with Transformers, removing hand-crafted components like anchors and NMS through end-to-end learning with bipartite matching.