DINOv3: Self-Supervised Learning at Scale

DINOv3 (Distillation with No Labels v3) is Meta's most powerful self-supervised vision model, trained on 1.7 billion images without any labels. At 7 billion parameters, it is the largest self-supervised vision model ever created, building on the foundational DINO framework of teacher-student self-distillation first introduced in 2021.

The DINO family's core mechanism is elegantly simple: a student network learns to match a teacher network's output distributions, where the teacher is an exponential moving average (EMA) of the student's own weights. No labels, no contrastive pairs, no negative sampling — just two views of the same image processed by student and teacher, with cross-entropy loss between their softmax output distributions. The original DINO (2021) demonstrated that this self-distillation produces ViT features with remarkable properties: attention maps that naturally segment objects, CLS tokens suitable for k-NN classification, and patch features that capture fine-grained spatial structure.

The key breakthrough in DINOv3 is Gram Anchoring — a new training technique that solves a critical scaling bottleneck: dense patch features degrading during long training schedules. In DINO and DINOv2, the CLS token representation improved continuously throughout training, but patch-level features (critical for segmentation, depth estimation, and detection) started degrading after ~100K iterations. The self-distillation objective optimizes primarily for global CLS agreement, causing patch features to collapse toward the global representation and lose spatial distinctiveness. Gram Anchoring adds a structural constraint on patch features by matching the student's patch-wise Gram matrix ($G = F^TF \in \mathbb{R}^{d \times d}$) to the teacher's, preserving the covariance structure of local features independently of the CLS token.

Key advances over DINOv2:

• 7B parameters (6x larger than DINOv2's 1.1B ViT-g) — the largest self-supervised vision model
• 1.7B images (12x more training data than DINOv2's LVD-142M dataset) with improved automatic curation
• Gram Anchoring — fixes dense feature degradation via patch-level structural loss, gaining +6 mIoU on ADE20k segmentation
• Two-stage high-resolution training — 90% at 224x224, then 10% at 518x518 with interpolated position embeddings for fine-grained spatial detail
• 88.4% ImageNet linear probe (self-supervised SOTA, +1.1% over DINOv2's 87.3%)
• 61.4 mIoU on ADE20k with frozen backbone (+6.2 over DINOv2, +13 over weakly-supervised models)

DINOv3 demonstrates for the first time that self-supervised models can outperform weakly-supervised models (like SigLIP, trained on billions of labeled image-text pairs) across a wide range of tasks. It achieves state-of-the-art on 60+ benchmarks with frozen backbones — meaning the same exact features, with no fine-tuning, set records on classification, segmentation, depth estimation, 3D understanding, video tracking, and more. The 7B model is then distilled into smaller deployable models (ViT-S through ViT-H+) that retain ~95% of performance at ~10% of the parameters.