DETR

The proposal/anchor paradigm and its accumulated complexity:

Every major object detector before DETR generates a massive number of candidate detections, then filters them down:

Faster R-CNN's proposal pipeline:

• Backbone produces features at 5 FPN levels (P2-P6)
• At each of ~200K spatial positions, 3 anchor ratios are evaluated
• RPN predicts objectness + box offsets for each of ~200K anchors
• NMS reduces to ~1000 proposals (IoU threshold 0.7)
• Stage 2 classifies and refines ~300 proposals
• Final NMS (IoU threshold 0.5) produces ~100 detections
• Hand-designed hyperparameters: anchor sizes (32, 64, 128, 256, 512), aspect ratios (0.5, 1.0, 2.0), NMS IoU thresholds (0.7 for RPN, 0.5 for final), positive/negative IoU thresholds (0.3/0.7), proposal counts (1000/300), score thresholds

Single-stage detectors (YOLO, SSD, RetinaNet):

• Dense anchors at every spatial position across multiple scales
• RetinaNet: 9 anchors per position x 5 levels = ~100K anchors for a 600x800 image
• Focal loss to handle extreme foreground/background imbalance (99.9% of anchors are background)
• NMS remains mandatory to remove duplicate detections

The fundamental problems:

1. NMS is non-differentiable: It is a hard selection operation that cannot be optimized during training. The model learns to produce overlapping detections, then relies on a separate heuristic to remove them.
2. Anchor design requires domain knowledge: The choice of anchor scales, ratios, and assignment thresholds significantly affects performance. RetinaNet's 9 anchor templates were tuned on COCO; different datasets may need different anchors.
3. No global reasoning about duplicate suppression: Each anchor/proposal is processed independently. The model has no mechanism to "know" that another anchor is already detecting the same object. NMS is a post-hoc fix for this lack of global coordination.
4. Two-stage overhead: RPN + NMS + RoIAlign + second-stage NMS adds both compute and engineering complexity. Each component has its own hyperparameters and failure modes.
5. Arbitrary prediction ordering: Detections are ordered by confidence score, which has no semantic meaning. There is no principled connection between the set of predicted objects and the set of ground-truth objects.

Object queries: N=100 learnable embeddings that directly produce the detection set:

DETR replaces the entire anchor/proposal/NMS pipeline with 100 learned vectors, each of dimension $d = 256$:

$Q = \{q_1, q_2, ..., q_{100}\} \in \mathbb{R}^{100 \times 256}$

These are randomly initialized and learned via backpropagation, just like any other network parameter. They are the only input to the decoder (besides the encoded image features).

How object queries work through the decoder:

Each of the 6 decoder layers performs three operations:

1. Self-attention among queries (queries reason about each other): $Q' = \text{MultiHeadSelfAttn}(Q + PE_q, Q + PE_q, Q)$

This is the key to NMS-free detection: queries can "see" what other queries are predicting and learn to avoid duplicates. If query 7 is already detecting a dog, query 23 learns (through self-attention) to detect a different object or output $\varnothing$.

2. Cross-attention to image features (queries look at the image): $Q'' = \text{MultiHeadCrossAttn}(Q' + PE_q, F_{enc} + PE_{2D}, F_{enc})$

where $F_{enc} \in \mathbb{R}^{HW \times 256}$ is the encoder output (850 tokens for a typical COCO image) and $PE_{2D}$ is the 2D positional encoding. Each query attends to all spatial positions and learns to focus on regions relevant to its predicted object.

3. FFN refinement (per-query feature refinement): $Q''' = \text{FFN}(Q'') = W_2 \cdot \text{ReLU}(W_1 \cdot Q'' + b_1) + b_2$

where the FFN has hidden dimension 2048 (8x expansion from $d = 256$).

After 6 decoder layers, each query $q_i$ produces a 256-dim output embedding that is fed to two parallel prediction heads:

$\hat{c}_i = \text{Linear}_{256 \rightarrow 92}(q_i^{(6)}) \quad \text{(class logits: 91 COCO classes + } \varnothing\text{)}$ $\hat{b}_i = \text{Sigmoid}(\text{MLP}_{256 \rightarrow 256 \rightarrow 256 \rightarrow 4}(q_i^{(6)})) \quad \text{(normalized box: } (x_c, y_c, w, h) \in [0,1]^4\text{)}$

Why N=100?

COCO images contain at most ~63 annotated objects (the maximum in the dataset). Setting N=100 provides comfortable headroom. The authors tested:

• N=50: 40.8 AP (some images with many objects lose detections)
• N=100: 42.0 AP (optimal)
• N=200: 41.8 AP (more unmatched queries provide weaker supervision signal per matched query)
• N=300: 41.5 AP (training becomes harder with many $\varnothing$ targets)

Query specialization (emergent behavior):

Despite having no explicit spatial initialization, trained queries develop clear specialization:

• Certain queries consistently detect objects in the bottom-left of images
• Other queries specialize in large objects (cars, buses) while others focus on small objects (bottles, cups)
• Some queries learn to detect specific object categories preferentially
• This specialization emerges purely from the set-based loss and cross-attention mechanism -- no hand-designed assignment rules

No NMS needed because:

1. Self-attention allows queries to coordinate and avoid duplicate detections
2. The Hungarian matching loss assigns each GT object to exactly one query, so the model is never rewarded for predicting the same object twice
3. Unmatched queries learn to output $\varnothing$ with high confidence, producing clean output without filtering