Multi-View Foundation Models

2D foundation models generalize remarkably well and lack 3D awareness entirely. Both facts follow from the same training setup: they were shown enormous numbers of single images and never asked whether two of them depicted the same point.

The failure is best understood as a broken invariant. Given views $i$ and $j$ of a scene and a 3D point $X$ projecting to $x^i$ and $x^j$, an ideal extractor would satisfy $F^i(x^i) \approx F^j(x^j)$. A single-image model has no term anywhere in its objective enforcing this — and no way to even express it, since it never sees both views at once. The invariant is not violated so much as absent. The paper's Figure 1 makes it concrete: a keypoint matched from one view to another lands correctly with MV-DINOv2 and visibly off with base DINOv2, and the error grows steadily with viewpoint angle.

The natural instinct is to fix this by scaling 3D pre-training, and that road is blocked. 3D data is scarce — large-scale datasets are limited in quantity and diversity, constrained by expensive acquisition, narrow category and environment coverage, and the difficulty of aligning geometry with texture and semantic labels. You cannot repeat the 2D recipe at 3D scale because the data does not exist. So the field lifts instead, optimizing a scene-level NeRF or Gaussian Splatting representation that aligns feature maps across views. Accurate, and per-scene — expensive and poorly scalable for anything large or interactive.

This paper's move is to notice that a consistent 3D model is a means, not the end. If what you want is consistent features, build those directly: let the views communicate inside the backbone and supervise them with correspondences from known camera poses. Doing so bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space — which is where the downstream tasks already live.