Object Detection2020

DETR

End-to-End Object Detection with Transformers

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko

Paper Overview

DETR (DEtection TRansformer) reformulates object detection as a direct set prediction problem, eliminating the need for hand-designed components like non-maximum suppression (NMS), anchor generation, or region proposal networks that have dominated detection architectures for a decade.

The architecture is elegantly simple: a ResNet-50 CNN backbone extracts a feature map of shape $H/32 \times W/32 \times 2048$ (e.g., $25 \times 34$ for a standard $800 \times 1066$ COCO image), which is projected to 256 channels, flattened to a sequence of $850$ tokens, and processed by a standard transformer encoder-decoder. The encoder (6 layers, 8-head self-attention, $d = 256$ ) performs global reasoning over all spatial positions. The decoder (6 layers) takes 100 learned object queries and transforms them into 100 output embeddings via cross-attention to the encoded image features. Two feed-forward network (FFN) heads predict a class label (92 COCO classes + "no object") and a normalized bounding box $(x_c, y_c, w, h) \in [0,1]^4$ for each query.

The key insight is that detection can be viewed as predicting a set of objects -- and the Hungarian algorithm provides a differentiable-compatible way to compute the optimal one-to-one matching between the $N = 100$ predictions and $M$ ground-truth objects (where typically $M \ll N$ ). Unmatched predictions are supervised to output the "no object" ( $\varnothing$ ) class. This set-based loss is permutation-invariant -- the model never needs to learn an ordering over detections.

Why this matters architecturally:

No anchors: Faster R-CNN uses 15 anchor templates per position across 5 FPN levels, generating ~200K candidates. DETR uses 100 learned queries -- period.
No NMS: The set loss naturally teaches queries to avoid predicting the same object (the Hungarian matching assigns each GT to exactly one query), so there are no duplicate detections to suppress.
No hand-tuned thresholds: IoU thresholds for anchor assignment (0.3/0.7), NMS thresholds (0.5), proposal counts (300) -- all eliminated.
Global reasoning: Transformer self-attention enables each spatial position to reason about the entire image, capturing long-range dependencies that CNNs miss (e.g., context: a tennis racket near a person on a court).

COCO benchmark results (ResNet-50 backbone, 500 epochs):

42.0 AP overall (vs. Faster R-CNN's 42.0 AP with the same backbone -- competitive)
20.5 AP-S on small objects (vs. Faster R-CNN's 24.1 AP-S -- the main weakness)
61.1 AP-L on large objects (vs. Faster R-CNN's 54.0 AP-L -- significant advantage)
Total parameters: 41M (ResNet-50: 23.5M backbone + 17.5M transformer + FFN heads)
Inference: 28 FPS on a V100 GPU (vs. Faster R-CNN's 26 FPS -- comparable speed, but no NMS latency variance)

DETR's strength is on large objects (where global attention shines) but weakness on small objects (where the $32\times$ downsampled feature map loses detail). Deformable DETR, introduced later, fixes this with multi-scale features and converges 10x faster.

Chapter Roadmap

Click any topic to jump in

Encoder-Decoder

Transformer backbone replacing region proposals and anchor boxes.

uses

Object Queries

Learned slot embeddings, one per potential object in the image.

Positional Encoding

2D sinusoidal encodings injecting spatial structure into attention.

Cross-Attention

Queries gather visual evidence from encoder features for localization.

uses

Hungarian Matching

Optimal one-to-one assignment of predictions to ground truth.

Set Prediction

Permutation-invariant loss removing the need for NMS.

uses

Deformable DETR

Sparse attention variant solving slow convergence and high resolution cost.

DETR replaces the entire proposal/anchor machinery of traditional detectors with a fixed set of N=100 learnable embeddings called object queries. Each query is a 256-dimensional vector that learns to specialize in detecting certain types of objects in certain spatial regions. Through 6 decoder layers of self-attention (among queries) and cross-attention (to image features), each query independently produces at most one detection or outputs 'no object'.

The Problem

The proposal/anchor paradigm and its accumulated complexity:

Every major object detector before DETR generates a massive number of candidate detections, then filters them down:

Faster R-CNN's proposal pipeline:

Backbone produces features at 5 FPN levels (P2-P6)
At each of ~200K spatial positions, 3 anchor ratios are evaluated
RPN predicts objectness + box offsets for each of ~200K anchors
NMS reduces to ~1000 proposals (IoU threshold 0.7)
Stage 2 classifies and refines ~300 proposals
Final NMS (IoU threshold 0.5) produces ~100 detections
Hand-designed hyperparameters: anchor sizes (32, 64, 128, 256, 512), aspect ratios (0.5, 1.0, 2.0), NMS IoU thresholds (0.7 for RPN, 0.5 for final), positive/negative IoU thresholds (0.3/0.7), proposal counts (1000/300), score thresholds

Single-stage detectors (YOLO, SSD, RetinaNet):

Dense anchors at every spatial position across multiple scales
RetinaNet: 9 anchors per position x 5 levels = ~100K anchors for a 600x800 image
Focal loss to handle extreme foreground/background imbalance (99.9% of anchors are background)
NMS remains mandatory to remove duplicate detections

The fundamental problems:

NMS is non-differentiable: It is a hard selection operation that cannot be optimized during training. The model learns to produce overlapping detections, then relies on a separate heuristic to remove them.
Anchor design requires domain knowledge: The choice of anchor scales, ratios, and assignment thresholds significantly affects performance. RetinaNet's 9 anchor templates were tuned on COCO; different datasets may need different anchors.
No global reasoning about duplicate suppression: Each anchor/proposal is processed independently. The model has no mechanism to "know" that another anchor is already detecting the same object. NMS is a post-hoc fix for this lack of global coordination.
Two-stage overhead: RPN + NMS + RoIAlign + second-stage NMS adds both compute and engineering complexity. Each component has its own hyperparameters and failure modes.
Arbitrary prediction ordering: Detections are ordered by confidence score, which has no semantic meaning. There is no principled connection between the set of predicted objects and the set of ground-truth objects.

The Solution

Object queries: N=100 learnable embeddings that directly produce the detection set:

DETR replaces the entire anchor/proposal/NMS pipeline with 100 learned vectors, each of dimension $d = 256$ :

$Q = \{q_1, q_2, ..., q_{100}\} \in \mathbb{R}^{100 \times 256}$

These are randomly initialized and learned via backpropagation, just like any other network parameter. They are the only input to the decoder (besides the encoded image features).

How object queries work through the decoder:

Each of the 6 decoder layers performs three operations:

1. Self-attention among queries (queries reason about each other): $Q' = \text{MultiHeadSelfAttn}(Q + PE_q, Q + PE_q, Q)$

This is the key to NMS-free detection: queries can "see" what other queries are predicting and learn to avoid duplicates. If query 7 is already detecting a dog, query 23 learns (through self-attention) to detect a different object or output $\varnothing$ .

2. Cross-attention to image features (queries look at the image): $Q'' = \text{MultiHeadCrossAttn}(Q' + PE_q, F_{enc} + PE_{2D}, F_{enc})$

where $F_{enc} \in \mathbb{R}^{HW \times 256}$ is the encoder output (850 tokens for a typical COCO image) and $PE_{2D}$ is the 2D positional encoding. Each query attends to all spatial positions and learns to focus on regions relevant to its predicted object.

3. FFN refinement (per-query feature refinement): $Q''' = \text{FFN}(Q'') = W_2 \cdot \text{ReLU}(W_1 \cdot Q'' + b_1) + b_2$

where the FFN has hidden dimension 2048 (8x expansion from $d = 256$ ).

After 6 decoder layers, each query $q_i$ produces a 256-dim output embedding that is fed to two parallel prediction heads:

$\hat{c}_i = \text{Linear}_{256 \rightarrow 92}(q_i^{(6)}) \quad \text{(class logits: 91 COCO classes + } \varnothing\text{)}$ $\hat{b}_i = \text{Sigmoid}(\text{MLP}_{256 \rightarrow 256 \rightarrow 256 \rightarrow 4}(q_i^{(6)})) \quad \text{(normalized box: } (x_c, y_c, w, h) \in [0,1]^4\text{)}$

Why N=100?

COCO images contain at most ~63 annotated objects (the maximum in the dataset). Setting N=100 provides comfortable headroom. The authors tested:

N=50: 40.8 AP (some images with many objects lose detections)
N=100: 42.0 AP (optimal)
N=200: 41.8 AP (more unmatched queries provide weaker supervision signal per matched query)
N=300: 41.5 AP (training becomes harder with many $\varnothing$ targets)

Query specialization (emergent behavior):

Despite having no explicit spatial initialization, trained queries develop clear specialization:

Certain queries consistently detect objects in the bottom-left of images
Other queries specialize in large objects (cars, buses) while others focus on small objects (bottles, cups)
Some queries learn to detect specific object categories preferentially
This specialization emerges purely from the set-based loss and cross-attention mechanism -- no hand-designed assignment rules

No NMS needed because:

Self-attention allows queries to coordinate and avoid duplicate detections
The Hungarian matching loss assigns each GT object to exactly one query, so the model is never rewarded for predicting the same object twice
Unmatched queries learn to output $\varnothing$ with high confidence, producing clean output without filtering

Key Points

N=100 learned 256-dim embeddings replace ~200K anchors, RPN, NMS, and all associated hyperparameters -- the entire proposal machinery reduced to a single parameter matrix

Self-attention among queries enables duplicate suppression without NMS: each query 'sees' what others are predicting and learns to output distinct objects or empty-set

Cross-attention to encoder output (HW x 256 tokens) allows each query to attend to any spatial position -- attention maps become sparse and object-focused after training

N=100 is optimal for COCO (max 63 objects per image): N=50 loses 1.2 AP from capacity limits, N=200 loses 0.2 AP from training difficulty with many empty-set targets

Query specialization emerges naturally: specific queries learn to detect objects in particular spatial regions, at particular scales, or of particular categories -- without any explicit assignment

Each query produces a 92-class prediction (91 COCO + empty-set) and a normalized box (x_c, y_c, w, h) via separate FFN heads after 6 decoder layers

Mathematical Formulation

Query Output (per decoder layer)

$q_i^{(l)} = \text{FFN}\left(\text{CrossAttn}\left(\text{SelfAttn}(q_i^{(l-1)}, Q^{(l-1)}),\; F_{enc}\right)\right)$

Each decoder layer transforms query q_i through three sub-layers: (1) self-attention with all other queries (for duplicate suppression), (2) cross-attention to the HW encoded image features (for localization), (3) FFN with 2048 hidden units (for refinement). After 6 layers, the final q_i is decoded into class logits and box coordinates.

Box Prediction Head

$\hat{b}_i = \sigma\left(\text{MLP}_{3\text{-layer}}(q_i^{(6)})\right) = (\hat{x}_c, \hat{y}_c, \hat{w}, \hat{h}) \in [0,1]^4$

Bounding boxes are predicted as normalized center coordinates and dimensions relative to the image. The sigmoid activation constrains all values to [0,1]. The 3-layer MLP has hidden dimension 256 with ReLU activations. This direct regression avoids the anchor-relative parameterization of Faster R-CNN.

Mathematical Intuition

Object queries are $N$ learned embeddings $q_i \in \mathbb{R}^d$ that act as 'slots' the decoder fills with object hypotheses. Each query attends to the encoder output via $\text{softmax}(q_i K^\top / \sqrt{d}) V$ , gathering features from spatial locations relevant to its slot. After training, each query specializes to a particular size/location prior — like learned anchor boxes but in feature space.

Back to All Papers

Study Plans

Computer Vision Study Plan All Papers Practice Problems

Object queries: N=100 learnable embeddings that directly produce the detection set:

DETR replaces the entire anchor/proposal/NMS pipeline with 100 learned vectors, each of dimension $d = 256$ :

$Q = \{q_1, q_2, ..., q_{100}\} \in \mathbb{R}^{100 \times 256}$

These are randomly initialized and learned via backpropagation, just like any other network parameter. They are the only input to the decoder (besides the encoded image features).

How object queries work through the decoder:

Each of the 6 decoder layers performs three operations:

1. Self-attention among queries (queries reason about each other): $Q' = \text{MultiHeadSelfAttn}(Q + PE_q, Q + PE_q, Q)$

2. Cross-attention to image features (queries look at the image): $Q'' = \text{MultiHeadCrossAttn}(Q' + PE_q, F_{enc} + PE_{2D}, F_{enc})$

3. FFN refinement (per-query feature refinement): $Q''' = \text{FFN}(Q'') = W_2 \cdot \text{ReLU}(W_1 \cdot Q'' + b_1) + b_2$

where the FFN has hidden dimension 2048 (8x expansion from $d = 256$ ).

After 6 decoder layers, each query $q_i$ produces a 256-dim output embedding that is fed to two parallel prediction heads:

Why N=100?

COCO images contain at most ~63 annotated objects (the maximum in the dataset). Setting N=100 provides comfortable headroom. The authors tested:

N=50: 40.8 AP (some images with many objects lose detections)
N=100: 42.0 AP (optimal)
N=200: 41.8 AP (more unmatched queries provide weaker supervision signal per matched query)
N=300: 41.5 AP (training becomes harder with many $\varnothing$ targets)

Query specialization (emergent behavior):

Despite having no explicit spatial initialization, trained queries develop clear specialization:

Certain queries consistently detect objects in the bottom-left of images
Other queries specialize in large objects (cars, buses) while others focus on small objects (bottles, cups)
Some queries learn to detect specific object categories preferentially
This specialization emerges purely from the set-based loss and cross-attention mechanism -- no hand-designed assignment rules

No NMS needed because:

Self-attention allows queries to coordinate and avoid duplicate detections
The Hungarian matching loss assigns each GT object to exactly one query, so the model is never rewarded for predicting the same object twice
Unmatched queries learn to output $\varnothing$ with high confidence, producing clean output without filtering

DETR

Paper Overview

Chapter Roadmap

Encoder-Decoder

Object Queries

Positional Encoding

Cross-Attention

Hungarian Matching

Set Prediction

Deformable DETR

Object Queries

The Problem

The Solution

Key Points

Mathematical Formulation

Query Output (per decoder layer)

Box Prediction Head

Bipartite Matching Loss

Encoder-Decoder Architecture

Cross-Attention for Object Localization

Set Prediction (Permutation Invariance)

Positional Encodings

Deformable DETR

Related Papers

Study Plans

DETR

Paper Overview

Chapter Roadmap

Encoder-Decoder

Object Queries

Positional Encoding

Cross-Attention

Hungarian Matching

Set Prediction

Deformable DETR

Object Queries

The Problem

The Solution

Key Points

Mathematical Formulation

Query Output (per decoder layer)

Box Prediction Head

Bipartite Matching Loss

Encoder-Decoder Architecture

Cross-Attention for Object Localization

Set Prediction (Permutation Invariance)

Positional Encodings

Deformable DETR

Related Papers

Study Plans