Paper breakdown3D Foundation ModelsFREE2025

Multi-View Foundation Models

Turning DINO, CLIP and SAM into Inference-Time Multi-View Consistent Variants

Leo Segre, Or Hirschorn, Shai Avidan

Concepts

Formulas

Figures

Paper Overview

Multi-View Foundation Models (Segre, Hirschorn & Avidan — Tel Aviv University, December 2025) starts from a mismatch that is easy to state and awkward to fix. A foundation model takes one RGB image and returns a feature map. Give it several views of the same scene and it processes each independently — so a single 3D point can receive materially different features depending on the angle it was photographed from. The features are excellent and the geometry is incoherent.

The established fix is to build a 3D representation — NeRF or Gaussian Splatting — and lift the features into it. That works and it is expensive: per-scene optimization, poor scalability, no good story for interactive use. FiT3D improved on this by predicting 3D-aware features in a single feed-forward pass, but it still processes one image at a time, so it has no mechanism for consistency across a set.

This paper takes a different route: keep everything in image space. Insert small multi-view adapters after each transformer block that let views attend to one another, condition them on camera geometry via Plücker ray embeddings, and train them with a correspondence loss. No 3D model is ever built. The result is a drop-in variant — MV-DINOv2, MV-CLIP, MV-SAM — that runs at inference time.

The numbers are unusually clean. On ScanNet++, DINOv2's location error falls from 0.1029 to 0.0247 while cosine similarity to the base model stays at 0.9376 — geometry fixed, semantics intact. The same recipe, unmodified, works on CLIP, DINOv3 and SAM.

Chapter Roadmap

Click any topic to jump in

Viewpoint Inconsistency

Foundation models see one image at a time — so the same 3D point gets different features from different angles.

fixed by

Multi-View Adapter

A 3D-attention + FFN + zero-conv block inserted after every transformer block, letting views talk to each other.

conditioned on

Plücker Ray Embeddings

Camera poses enter as a raymap — per-pixel ray origin and direction, relative to the first view.

supervised by

Correspondence Training

A naive L2 consistency loss collapses to a constant. SoftArgMax over a similarity map fixes it.

trained via

Training Recipe

LoRA r=32 on the backbone, full training on the adapters, COLMAP correspondences, 230 ScanNet++ scenes.

evaluated by

Evidence and caveats

Results

Location error drops 4-8x across DINOv2, CLIP, DINOv3 and SAM — with no architectural change between them.

The Regularization Trade-off

Dropping regularization gives the *best* geometry and destroys the semantics — the paper's sharpest result.

Concept by concept

7 sections, in reading order.

01 / 07The Viewpoint Inconsistency Problem

2D foundation models generalize remarkably well and lack 3D awareness entirely. Both facts follow from the same training setup: they were shown enormous numbers of single images and never asked whether two of them depicted the same point.

The failure is best understood as a broken invariant. Given views $i$ and $j$ of a scene and a 3D point $X$ projecting to $x^i$ and $x^j$ , an ideal extractor would satisfy $F^i(x^i) \approx F^j(x^j)$ . A single-image model has no term anywhere in its objective enforcing this — and no way to even express it, since it never sees both views at once. The invariant is not violated so much as absent. The paper's Figure 1 makes it concrete: a keypoint matched from one view to another lands correctly with MV-DINOv2 and visibly off with base DINOv2, and the error grows steadily with viewpoint angle.

The natural instinct is to fix this by scaling 3D pre-training, and that road is blocked. 3D data is scarce — large-scale datasets are limited in quantity and diversity, constrained by expensive acquisition, narrow category and environment coverage, and the difficulty of aligning geometry with texture and semantic labels. You cannot repeat the 2D recipe at 3D scale because the data does not exist. So the field lifts instead, optimizing a scene-level NeRF or Gaussian Splatting representation that aligns feature maps across views. Accurate, and per-scene — expensive and poorly scalable for anything large or interactive.

This paper's move is to notice that a consistent 3D model is a means, not the end. If what you want is consistent features, build those directly: let the views communicate inside the backbone and supervise them with correspondences from known camera poses. Doing so bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space — which is where the downstream tasks already live.

The Viewpoint Inconsistency Problem diagram

Key Points

1
A foundation model maps one RGB image to a feature map. Given multiple views of one scene it runs independently per image — nothing ties the outputs together
2
Consequence: the same 3D point does not reliably get the same feature across views. Matching by feature similarity drifts
3
Error grows with viewpoint angle: DINOv2 climbs from ~0.017 to ~0.035 as angular separation goes 0° → ~105°, while MV-DINOv2 stays nearly flat near ~0.005
4
This matters exactly where 3D work happens — orbit captures, aerial imagery, and sparse-view datasets all involve wide baselines where local feature similarity breaks down
5
Scaling 3D pre-training is not an option: 3D data is limited in quantity and diversity, with expensive acquisition and hard geometry-semantics alignment
6
NeRF/Gaussian-Splatting lifting works but requires per-scene optimization — expensive and poorly scalable
7
The paper separates semantic correspondence (all pixels of an object match) from geometric correspondence (a specific point matches that specific point) — a tension noted by Probe3D — and targets the geometric kind

Key Points

1
An adapter $A_\phi$ is inserted after each transformer block $f_\theta$ — geometry-aware bridges enabling cross-view communication at every feature level, not just the last
2
Internals are minimal: 3D Attention → FFN → Zero-Conv, with a residual connection around the whole block
3
Adapters are initialized as zero-convolution layers, so at step zero the augmented model is mathematically identical to the original pretrained transformer
4
The adapter operates on latent feature maps for multiple views jointly, integrating geometric context from the relative camera poses
5
Because the intervention is between blocks, the framework applies to any transformer-based foundation model without architectural changes
6
Everything happens at inference time: no per-scene optimization, no NeRF, no Gaussian Splatting, no rendering

Mathematical Formulation

Adapter Insertion

$\tilde{z}^l_i = f_\theta\big(z^{l-1}_i\big), \qquad z^l_{1:M} = A_\phi\big(\tilde{z}^l_{1:M}\big)$

The pretrained block $f_\theta$ runs per view $i$ — it cannot see the others. The adapter $A_\phi$ then runs across all $M$ views together, which is the only point where cross-view information mixes. Note the subscript change from $i$ to $1{:}M$ : that is the entire architectural idea in one line.

Key Points

1
Camera poses are represented with a Plücker embedding: each pixel encodes a ray origin and direction in 3D space
2
The raymap has the same spatial dimensions as the latent feature map, so it embeds through a parallel Ray Patch Embedding and concatenates with the image patch embedding
3
Rays are computed relative to the camera pose of the first conditional view — relative, not absolute, so the model never depends on a global world frame
4
This encoding lets the model reason about geometric correspondences across viewpoints rather than inferring them from appearance alone
5
Camera information is optional — the paper reports strong multi-view consistency even without it
6
The ablation prices it precisely: adding Plücker moves DINOv2's location error from 0.0262 to 0.0247, the smallest of the three contributions

Mathematical Formulation

Plücker Raymap

$p(u,v) = \big[\, o(u,v),\; d(u,v) \,\big]$

For each pixel $(u,v)$ , $o$ is the ray origin and $d$ the ray direction in 3D, both computed relative to the pose of the first conditional view. The resulting raymap matches the latent feature map's spatial dimensions and is patch-embedded alongside the image.

Key Points

1
The naive loss $\mathcal{L}_{\text{naive}} = \lVert F^i(x^i_l) - F^j(x^j_m) \rVert_2$ is globally minimized by the trivial solution $F^i(x) = F^j(x) = C$ — feature collapse
2
The fix (adopted from Zhang et al.) is a geometry-aware dense loss that never compares two features directly
3
Instead: compute a similarity map between the normalized query feature $F^i(x^i_l)$ and all spatial locations in the target map $F^j$
4
Softmax the similarity map with temperature $\tau$ , then take a differentiable SoftArgMax to get a predicted correspondence location
5
Penalize the Euclidean distance between predicted and ground-truth locations — a loss on where the match lands, not on vector similarity
6
This prevents trivial collapse: a constant feature map yields a uniform similarity map, whose SoftArgMax lands in the image centre and scores terribly
7
A regularization term keeps refined features near the original space — high cosine similarity and similar norm to the base feature
8
The loss is computed over multiple corresponding points and multiple view pairs per batch, using correspondences derived from known camera poses and ray geometry

Mathematical Formulation

The Naive Loss (and why it fails)

$\mathcal{L}_{\text{naive}} = \big\lVert F^i(x^i_l) - F^j(x^j_m) \big\rVert_2$

Minimized globally by $F^i(x) = F^j(x) = C$ for any constant $C$ . The network learns to output the same feature everywhere, achieves zero loss, and is worthless. This is the equation the rest of the section exists to avoid.

SoftArgMax Correspondence Prediction

$p^{i \to j}_l = \text{softmax}\!\left(\frac{S^{i \to j}_l}{\tau}\right), \qquad \hat{x}^j_m = \sum_{u \in \Omega^j} p^{i \to j}_l(u)\, u$

$S^{i \to j}_l$ is the similarity map between the query feature in view $i$ and every location in view $j$ . Softmax with temperature $\tau$ turns it into a distribution over locations; the expectation under that distribution is a differentiable predicted position $\hat{x}^j_m$ .

Geometry-Aware Dense Loss

$\mathcal{L}_{\text{corr}} = \big\lVert x^j_m - \hat{x}^j_m \big\rVert_2$

Euclidean distance between the ground-truth correspondence (projected from 3D via COLMAP) and the predicted one. The gradient flows back through SoftArgMax into the similarity map and hence into the features — but the penalty is on position, never on raw feature distance.

Feature-Space Regularization

$\mathcal{L}_{\text{norm}} = 1 - \frac{\lVert F^k \rVert}{\lVert \tilde{F}^k \rVert}, \qquad \mathcal{L}_{\text{angle}} = 1 - \cos\big(F^k, \tilde{F}^k\big)$

$F^k$ is the original frozen feature and $\tilde{F}^k$ the adapted one. The two terms hold the refined features at a similar norm and a similar direction to the pretrained representation, preserving generalization while geometry is added.

Total Objective

$\mathcal{L}_{\text{reg}} = \sum_k \big(\mathcal{L}_{\text{norm}} + \mathcal{L}_{\text{angle}}\big), \qquad \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{corr}} + \lambda_{\text{reg}} \mathcal{L}_{\text{reg}}$

$\lambda_{\text{reg}}$ controls the balance. As the ablation shows, this single scalar mediates the paper's central trade-off: geometric accuracy versus fidelity to the base model.

Key Points

1
COLMAP reconstructs camera poses and a sparse 3D point cloud per scene; those 3D points are projected back into all associated views to yield dense pixel-level correspondences
2
This supervision is free of manual labels and does not require adaptation for each model — the same pipeline trains every backbone
3
Training data: ScanNet++, 230 scenes / 140K views
4
Schedule: 24 epochs (12 for SAM and DINOv3); 1,000 sets of 4 images at 512×512 per scene per epoch; 128 sampled correspondences per iteration
5
Optimizer: AdamW, lr $10^{-4}$ , linear scheduler. LoRA on the backbone with rank $r{=}32$ , $\alpha{=}32$ ; adapter parameters $\phi$ are fully fine-tuned
6
Backbones tested: DINOv2 ViT-S/14 with register tokens, SAM ViT-B, DINOv3 ViT-B/16, CLIP ViT-B/16
7
Evaluation: 50 held-out ScanNet++ scenes, plus a generalization set of 115 scenes from 8 datasets — MipNeRF-360, LLFF, Tanks & Temples, NeRF-Synthetic, ScanNet, Objaverse, Nerfbusters, NeRF2NeRF
8
Normals probe: backbone frozen; a lightweight self-attention head predicts per-pixel normals in a shared frame (camera 0 as origin), outputting 4 channels $(x,y,z,\sigma)$ with an uncertainty-weighted loss; AdamW, 50 epochs, lr $5\times10^{-5}$

Key Points

1
Location Error measures geometric consistency, normalized to image size — 0.01 is roughly 5 pixels on a 512×512 image
2
Similarity to Base measures semantic preservation — critical, because naive fine-tuning could achieve low location error by memorizing correspondences while destroying semantic understanding
3
DINOv2 / ScanNet++: $0.1029 \rightarrow \mathbf{0.0247}$ (4.2x), Base Sim 0.9376
4
DINOv2 / Generalization: $0.1404 \rightarrow \mathbf{0.0782}$ , Base Sim 0.8003 — trained only on indoor ScanNet++, still holds across 8 diverse datasets
5
CLIP / ScanNet++: $0.1505 \rightarrow \mathbf{0.0301}$ (5.0x), Base Sim 0.8824
6
DINOv3 / ScanNet++: $0.1025 \rightarrow \mathbf{0.0314}$ , Base Sim 0.9411 — a newer backbone, no architectural changes
7
SAM / ScanNet++: $0.1846 \rightarrow \mathbf{0.0244}$ — a 7.6x reduction, the largest — with Base Sim 0.9547
8
vs FiT3D (DINOv2/ScanNet++): FiT3D reaches 0.0858 at Base Sim 0.7484 — this method is 3.5x more accurate and far more faithful semantically
9
FiT3D + Base (0.1062) is worse than FiT3D alone (0.0858) — naively combining representations hurts
10
Surface normals on NAVI (frozen backbones): RMSE $56.00$ (DINOv2) / $51.00$ (FiT3D) → 32.17; [email protected]° $0.156/0.176 \rightarrow$ 0.251; recall@30° $0.434/0.494 \rightarrow$ 0.693
11
Because the backbone stays frozen during probing, the probe does not learn 3D reasoning — it merely reads out structure already present in the features
12
MV-SAM: a single click in one view yields consistent segmentation masks across all views; base SAM fails to maintain correspondence

Key Points

1
Base DINOv2: 0.1029 location error (no base-similarity figure — it is the base)
2
(+) LoRA training: $0.1029 \rightarrow 0.0347$ , Base Sim 0.9117 — the correspondence objective alone does most of the geometric work, but is limited to single-view processing
3
(+) Multi-View Adapter: $0.0347 \rightarrow 0.0262$ , Base Sim $0.9117 \rightarrow 0.9360$ — improves geometry and semantic alignment simultaneously, so it is not a trade-off
4
(+) Plücker (full method): $0.0262 \rightarrow 0.0247$ , Base Sim 0.9376
5
(−) Regularization: location error improves to 0.0204 — the best in the table — while Base Sim collapses from 0.9376 to 0.1054
6
The final row is the paper's thesis stated as a counterexample: geometric accuracy is trivially purchasable by destroying the representation, so it cannot be the only thing you measure
7
The final design trades ~20% worse location error for ~9x better semantic fidelity — and the paper is explicit that this is a deliberate balance

Go deeper

Study Plans

3D CV Study Plan Computer Vision Study Plan All Papers Practice Problems