Loading...
Efficient Universal Perception Encoder
Chenchen Zhu, Saksham Suri, Cijo Jose, Maxime Oquab, Marc Szafraniec, Wei Wen, Yunyang Xiong, Patrick Labatut, Piotr Bojanowski, Raghuraman Krishnamoorthi, Vikas Chandra
Read the Paper on arXivEdge devices need vision encoders that are small (M params), fast ( ms on a phone CPU), and universal — strong at image classification, dense prediction (segmentation, depth, correspondence), and vision-language modeling, all from a single backbone. Domain experts achieve only one or two of these: PEcore and SigLIP2 win at classification and VLM but lose dense prediction; DINOv3 wins dense prediction but lacks language alignment.
The obvious fix is agglomerative distillation — distill from several expert teachers into one student — but prior work like RADIO and DUNE shows this falls apart at edge sizes. RADIOv2.5-B sits multiple points behind the best per-task expert on dense prediction and VLM. The reason is capacity: an 86 M-parameter student does not have enough room to absorb the conflicting feature distributions of 3+ specialized teachers directly.
EUPE (Meta, arXiv:2603.22387) introduces a deceptively simple recipe: scale up before scaling down. Stage 1 distills three foundation teachers — PEcore-G (1.9B, image-text), PElang-G (1.7B, language-aligned vision), and DINOv3-H+ (840M, dense prediction) — into a single 1.9 B-parameter proxy that has enough capacity to unify their representations. Stage 2 then distills that single unified proxy into the efficient student at fixed 256×256 resolution for 390 k iterations. Stage 3 finishes with a short multi-resolution finetune over a {256, 384, 512} pyramid for 100 k iterations.
The payoff: EUPE-ViT-B (86 M params, 47 GFLOPs, 55 ms on iPhone 15 Pro CPU at 256×256) matches or beats every same-size domain expert on its home task. ImageNet-zero-shot 79.7% beats PEcore-B (78.4) and SigLIP2-B (78.2). ImageNet-KNN 84.1% beats SigLIP2-B (83.2) and DINOv3-B (83.0). ADE20k mIoU 52.4 beats DINOv3-B (51.8). SPair correspondence 51.3 matches DINOv3-B (51.3). Versus the agglomerative baseline RADIOv2.5-B, EUPE-ViT-B wins by +15 points on SPair, +3.4 on ADE20k, +3.4 on TextVQA, +1.5 on GQA. The full family ships at six points on the size frontier: ViT-T (6 M), ViT-S (21 M), ViT-B (86 M), and ConvNext-T (29 M), -S (50 M), -B (89 M).
The ablation tells the real story. Without the proxy — i.e. distilling all three teachers directly into the 86 M student (Stage 2 only) — SPair collapses to 35.1 and ADE20k to 41.9. Add Stage 1 and you reach 41.0 / 43.3. Add Stage 3 multi-res on top and you finally hit 51.3 / 52.4. The proxy is not optional cleanup; it's load-bearing.
Click any topic to jump in
The core principle: small students can't absorb multiple teachers directly, so first concentrate knowledge into a 1.9B proxy and only then distill down.
Pipeline
Three complementary foundation teachers (PEcore-G, PElang-G, DINOv3-H+) distill into the proxy via per-teacher adapter heads + frozen mean-std normalization.
Long fixed-256 distillation from the proxy (390k iters) followed by a short {256,384,512}-pyramid finetune (100k iters) with independent scale sampling.
Six checkpoints (ViT-T/S/B + ConvNext-T/S/B) on the latency-accuracy frontier. EUPE-ViT-B beats every same-size domain expert in 55 ms on iPhone 15 Pro CPU.
EUPE's central thesis: efficient students cannot directly absorb the conflicting feature distributions of multiple specialized teachers. The fix is to first concentrate everything into one large proxy that can unify the knowledge, and only then distill that single coherent teacher into the small student.
Agglomerative distillation methods like AM-RADIO and DUNE distill from several expert teachers in parallel: a single student sees losses from CLIP-style, DINO-style, and SAM-style teachers in the same backward pass. This works at 300 M+ student size, where the student has enough capacity to host all three feature geometries, but collapses at the edge-deployable scale of 6–100 M parameters.
The failure mode is visible across every task domain. RADIOv2.5-B (a state-of-the-art 86 M agglomerative encoder) trails its DINOv3-B dense-prediction teacher by 2.6 points on SPair (48.7 vs 51.3) and 2.8 points on ADE20k (49.0 vs 51.8). It trails the PEcore-B image-understanding teacher by 3.8 points on ImageNet-ZS (74.6 vs 78.4). DUNE-B drops out of zero-shot classification entirely (no text encoder).
The EUPE authors confirm the capacity diagnosis with a clean ablation. They run a "Stage 2 only" variant: skip the proxy, distill the same three teachers (PEcore-G, PElang-G, DINOv3-H+) directly into the ViT-B student. The result is catastrophic on dense prediction:
| Setup | SPair | NYUv2 (RMSE↓) | ADE20k |
|---|---|---|---|
| Stage 2 only (direct multi-teacher) | 35.1 | 0.616 | 41.9 |
| Stage 1 + 2 (proxy + fixed-res) | 41.0 | 0.557 | 43.3 |
| Stage 1 + 3 (proxy + multi-res) | 53.3 | 0.388 | 52.0 |
| Full pipeline (1 + 2 + 3) | 51.3 | 0.391 | 52.4 |
Direct distillation gives away 16 points of SPair accuracy and nearly doubles depth RMSE versus the full pipeline. The student is not refusing to learn — it is being pulled in three directions at once and ends up as a poor average of all three.
EUPE replaces direct multi-teacher distillation with a two-step indirection through a proxy teacher that has enough capacity to absorb every expert's representation losslessly:
Stage 1 — Scale up. A 1.9 B-parameter ViT-G proxy is trained to mimic three foundation teachers in parallel. Each teacher contributes a class-token loss + patch-token loss; the proxy's adapter heads align dimensions per teacher. This is RADIO-style multi-teacher distillation, but the student here is huge (1.9 B), so the capacity-mismatch problem disappears. The proxy ends up with one unified feature geometry that respects all three experts (Table 4: proxy hits IN1k-ZS 84.8, ADE20k 55.9, SPair 53.8 — competitive with the experts themselves).
Stage 2 — Scale down (fixed resolution). The 1.9 B proxy is then frozen and used as the only teacher for the efficient student (ViT-B/S/T or ConvNext-B/S/T). Training runs at 256×256 for 390 k iterations, batch 8192, base LR , weight decay . Single-teacher distillation is much easier — there is one loss and one target geometry — so the small student converges cleanly.
Stage 3 — Scale down (multi-resolution). A short 100 k-iteration finetune at batch 4096, LR , with the input resampled from a {256, 384, 512} pyramid. Student and teacher independently sample a scale per iteration, exposing the student to the proxy's behaviour at every resolution downstream tasks care about.
The ablation in the table above shows each stage carries weight. Stage 1 alone (no multi-res) reaches 41.0 SPair — already +6 over direct multi-teacher. Adding Stage 3 jumps to 51.3 SPair and 52.4 ADE20k, matching DINOv3-B on its home turf while staying universal. The total recipe is 490 k iterations of student training plus the proxy pretrain — modest by foundation-model standards, and entirely label-free (uses LVD-1689M images).
Direct multi-teacher distillation into an 86 M student gives SPair=35.1 and ADE20k=41.9 — far below DINOv3-B (51.3 / 51.8). The student lacks capacity to host three teacher geometries at once.
Inserting a 1.9 B proxy as an intermediate "Teaching Assistant" closes the gap: 41.0 / 43.3 with Stages 1+2, and 51.3 / 52.4 with the full 1+2+3 pipeline.
The proxy carries 22× more parameters than the student (1.9 B vs 86 M), which is exactly the gap that makes direct multi-teacher distillation feasible at the proxy level but infeasible at the student level.
Stage 1 trains the proxy from three frozen experts; Stages 2&3 train the student from the proxy only — never directly from the original experts.
All three stages are label-free: training uses the DINOv3 dataset (LVD-1689M) with 10% sampling probability for ImageNet1k.
A 7 B proxy was tried and hurts student VLM scores (Table 8): the proxy-to-student capacity gap becomes too large. The authors leave progressive Teacher-Assistant chaining (Mirzadeh et al., 2020) as future work.
Class-token loss is pure cosine similarity. Patch-token loss is α=0.9 cosine plus β=0.1 smooth-L1 against the normalized teacher tokens. z denotes the student's adapter-projected output; ȳ denotes the teacher's mean-std normalized tokens.
Stage 1 sums three per-teacher losses for the proxy. Stages 2 & 3 reduce to a single term where the proxy is the only teacher — no per-teacher weighting needed.
The student has under 2% of the combined teacher parameters. Going through a proxy raises the effective capacity ratio at each distillation step (proxy/teachers ≈ 43%, student/proxy ≈ 4.5%) — neither step asks for a 50× compression.
An 86 M student has roughly of the combined parameter budget of its three teachers. There is not enough representational capacity for it to host three disjoint feature geometries simultaneously. Compress the teachers into one 1.9 B proxy first and the student only has to track a single, already-reconciled distribution.