Loading...
Real-Time End-to-End Object Detection
Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, Guiguang Ding
Read the Paper on arXivYOLOv10 (Tsinghua University, 2024) achieves true end-to-end real-time object detection by eliminating Non-Maximum Suppression (NMS) — the last non-differentiable post-processing step in the detection pipeline. It does this through consistent dual assignments: training with both one-to-many (rich supervision) and one-to-one (NMS-free) branches simultaneously, using only the one-to-one branch at inference.
Why NMS elimination matters: NMS adds 1-2ms latency per image on GPU (more on edge devices), varies with scene complexity (more objects → slower), is non-differentiable (can't be optimized during training), and complicates deployment (requires NMS library support). Previous end-to-end detectors (DETR, RT-DETR) eliminated NMS but used expensive Transformer decoders, making them slower overall.
Beyond NMS-free training, YOLOv10 introduces a holistic efficiency-accuracy design with multiple architectural innovations:
| Innovation | Savings | Where Applied |
|---|---|---|
| Lightweight classification head | 40% fewer head params | All models |
| Spatial-channel decoupled downsampling | Better info preservation | All models |
| Rank-guided block design | Stage-specific block selection | M/B/L/X |
| Large-kernel depthwise convolutions | Larger receptive field | L/X |
| Partial Self-Attention (PSA) | Global context at lower cost | S/M/B/L/X |
Key results on COCO val2017:
Click any topic to jump in
Joint optimization of latency and accuracy across the entire architecture.
One-to-many head trains, one-to-one head deploys — sharing weights for consistency.
Compact inverted bottleneck reducing FLOPs in the backbone.
Self-attention applied to half the channels for cheap global context.
Inference with no post-processing — predictions go straight to output.
Family of variants from nano to extra-large with compound scaling.
Outperforms RT-DETR at same AP with lower latency on edge GPUs.
The core innovation of YOLOv10 is training with two label assignment strategies simultaneously — one-to-many for rich supervision and one-to-one for NMS-free inference — while enforcing consistency between them to ensure the one-to-one branch benefits from the many-to-one branch's richer training signal.
The one-to-many vs one-to-one dilemma:
One-to-many assignment (used in YOLOv5-v8, FCOS, ATSS): Each ground truth object is assigned to multiple predictions (typically 5-20) based on IoU overlap and center proximity. This provides rich supervision — many gradient signals per object per training step, leading to fast convergence and strong accuracy. But: Multiple predictions per object means redundant detections at inference, requiring NMS.
One-to-one assignment (used in DETR, Deformable DETR): Hungarian matching assigns exactly one prediction per ground truth. No redundancy, no NMS needed at inference. But: Only one gradient signal per object per step — weak supervision leads to slow convergence and lower accuracy. DETR required 500 epochs to converge (vs 300 for YOLO).
Previous attempts at NMS-free YOLO (end2end variants) simply replaced one-to-many with one-to-one — this caused significant accuracy drops (2-3 mAP) because the weak supervision couldn't match the one-to-many training quality.
The question: Can we get the training benefits of one-to-many supervision while deploying the NMS-free one-to-one branch at inference?
Consistent Dual Assignments uses both strategies simultaneously:
Architecture: Two parallel detection heads sharing the same backbone and neck features:
Both heads share the same backbone and FPN neck — they differ only in their assignment strategy and lightweight head layers. This sharing is critical: the one-to-one head benefits from features trained by the richer one-to-many supervision.
Consistency regularization (): Ensures the one-to-one head's selected predictions are consistent with the one-to-many head's top predictions. Specifically, for each ground truth, the one-to-one head's assigned prediction should have high alignment score in the one-to-many head's metric. This prevents the two heads from diverging during training.
Training vs Inference:
The result: The one-to-one head achieves accuracy comparable to the one-to-many head (which would need NMS), because it was trained alongside richer supervision that shaped the shared features.
One-to-many assignment provides rich supervision but creates redundant predictions
One-to-one assignment (Hungarian matching) assigns exactly one prediction per object
Both branches share the same feature backbone for consistency
Training uses both branches; inference uses only one-to-one (no NMS needed)
Reduces inference latency by eliminating post-processing
The total loss combines one-to-many (o2m) and one-to-one (o2o) losses with weighting factor lambda
Dual heads share weights but use different label assignment functions: assigns each GT to its top- predictions (rich gradient signal during training), while assigns each GT to a single best prediction (deployable without NMS). The consistency objective minimizes over predictions , so the one-to-one head inherits the supervision quality of the one-to-many head while staying NMS-free at inference.