Paper breakdownVision-LanguageFREE2025

InternVL3.5

Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei,

Read the Paper on arXiv

Concepts

Formulas

Figures

Demos

Paper Overview

InternVL3.5 (Wang, Gao, Gu et al. — InternVL Team, Shanghai AI Laboratory, 2025) is an open family of multimodal models from 1B to 241B parameters, dense and mixture-of-experts. Its main argument is about recipes rather than architecture. The model keeps the familiar ViT–MLP–LLM layout, and the paper attributes its gains to three changes in how the model is trained and served.

The first is Cascade RL. Offline RL (MPO) is cheap and stable but plateaus early, while online RL (GSPO) climbs higher but is expensive and fragile from a weak start. Running MPO first as a warm-up and GSPO second as a refinement gives the 8B model 60.3 on a seven-benchmark reasoning suite. Two full episodes of GSPO alone reach 58.2 while using about twice the GPU hours.

The second and third target inference cost. A Visual Resolution Router (ViR), trained through Visual Consistency Learning (ViCO), decides per image patch whether 64 visual tokens are enough or 256 are needed, which halves visual tokens at almost no loss in accuracy. Decoupled Vision-Language Deployment (DvD) runs the vision encoder and the LLM on separate servers in an asynchronous pipeline so neither blocks the other. Together they give up to 4.05x request throughput over the baseline.

The paper reports up to +16.0% overall reasoning over InternVL3, 77.7 on MMMU for the 241B-A28B flagship, a general-multimodal score of 74.1 vs GPT-5's 74.0, and new capabilities in GUI control, embodied spatial reasoning and SVG generation, with all weights released.

Explainer videoThe whole paper, animated

Chapter Roadmap

Click any topic to jump in

Interactive demo

The Open-Source Gap

Reasoning, text and agentic tasks are where open MLLMs trail. Unstable RL and rising visual-token cost are the two obstacles.

answered by

ViT–MLP–LLM Family

InternViT + Qwen3/GPT-OSS, nine sizes from 1B to 241B-A28B. Dynamic tiling with 256 tokens per tile after pixel shuffle.

trained via

Native Pre-Training + SFT

~250B tokens of joint text + multimodal pre-training, then ~130B tokens of SFT with Thinking-mode and agentic data.

then optimised for

Reasoning and efficiency

Cascade RL

Offline MPO warm-up, then online GSPO: 60.3 vs 58.2 for two GSPO episodes, at about half the GPU hours.

ViR + ViCO

A per-patch router picks 64 or 256 tokens, trained by consistency KL then self-labelled loss ratios. 50% fewer tokens.

used at inference

Inference-time compute and serving

Test-Time Scaling

Thinking mode for depth plus Best-of-N with VisualPRM for breadth. Applied to reasoning tasks only.

Decoupled Deployment

ViT and LLM on separate servers in an async pipeline: up to 2.01x alone and 4.05x with ViR.

produces

Outcomes

Results

74.1 vs GPT-5's 74.0 on general tasks, 77.7 MMMU, and Cascade RL gains at every scale.

GUI, Embodied, SVG

92.9 ScreenSpot-v2, 69.5 VSI-Bench and 70.7 SGP-Bench: one open model across agentic domains.

Concept by concept

9 sections, in reading order.

01 / 09The Open-Source Gap

By mid-2025 multimodal models were no longer judged on describing pictures. The tasks that mattered had moved to reasoning (MMMU, MathVista), text-heavy work (AIME, GPQA, MMLU-Pro) and agentic behaviour (GUI control, embodied spatial reasoning). On those, commercial models had opened a wide lead over open-source ones, and the InternVL team frames InternVL3.5 as an attempt to close it.

The paper names two obstacles. The first is reinforcement learning. Open-source groups had been applying RL to MLLMs to buy reasoning, but in the paper's words a stable, effective, and scalable RL framework for MLLMs was still an open problem. Offline methods such as DPO are cheap and stable but have a lower ceiling; online methods such as PPO or GRPO reach higher but cost far more GPU time and are touchier to train.

The second obstacle is cost at inference. High-resolution understanding, multi-image input and long video all multiply the number of visual tokens, and each extra token makes deployment slower. The capabilities that close the quality gap widen the cost gap.

InternVL3.5 answers each obstacle with its own mechanism. Cascade RL runs offline RL as a warm-up and then online RL as a refinement, aimed at reasoning. A Visual Resolution Router (ViR) decides per image patch how hard the visual tokens can be compressed, and Decoupled Vision-Language Deployment (DvD) puts the vision encoder and the LLM on separate GPUs so they stop blocking each other. Both target efficiency. Against InternVL3 the paper reports up to +16.0% overall reasoning performance and a 4.05x inference speed-up. Against GPT-5, the flagship InternVL3.5-241B-A28B narrows the overall gap to 3.9%.

Key Points

1
Frontier MLLM evaluation has shifted to reasoning, text and agentic tasks, where commercial models led open-source ones by wide margins
2
Open problem #1: no stable, effective, scalable RL recipe for MLLMs. Offline RL (DPO-style) is cheap but has a low ceiling, while online RL is strong but expensive
3
Open problem #2: high-resolution, multi-image and video inputs multiply visual tokens, which drives up inference cost
4
Three proposals: Cascade RL for reasoning, plus ViR and DvD for efficiency
5
Headline vs InternVL3: up to +16.0% overall reasoning and 4.05x inference speed-up
6
InternVL3.5-241B-A28B narrows the overall gap to GPT-5 to 3.9%, and on general multimodal tasks scores 74.1 vs 74.0 (Table 2)
7
Every model from 1B to 241B, dense and MoE, is released with code

Key Points

1
Same ViT–MLP–LLM paradigm as earlier InternVL versions; the gains come from training and serving, not a new layout
2
LLM initialised from Qwen3 (0.6B–235B-A22B) or GPT-OSS-20B; vision encoder is InternViT-300M or InternViT-6B
3
Nine sizes: six dense (1B, 2B, 4B, 8B, 14B, 38B) and three MoE (20B-A4B, 30B-A3B, 241B-A28B). The 241B-A28B model has 240.7B total and 28B activated parameters
4
Dynamic High Resolution: match the closest predefined aspect ratio, tile into $448 \times 448$ patches, and add a thumbnail for global context
5
Each tile is 1024 ViT tokens, compressed by pixel shuffle to 256 tokens before the LLM
6
Tiling adapts to image shape, not content. Closing that gap is what the ViR in InternVL3.5-Flash is for

Mathematical Formulation

Pixel Shuffle Token Budget

$\underbrace{32 \times 32}_{1024\ \text{ViT tokens}} \;\xrightarrow{\ \text{pixel shuffle } (2\times 2)\ }\; \underbrace{16 \times 16}_{256\ \text{LLM tokens}}, \qquad \xi = \tfrac{256}{1024} = \tfrac{1}{4}$

Pixel shuffle trades spatial resolution for channel depth: every $2 \times 2$ block of ViT features is concatenated into one token. The paper calls this a compression rate of $\xi = \tfrac{1}{4}$ . InternVL3.5-Flash adds a second, more aggressive shuffle with $\xi = \tfrac{1}{16}$ (64 tokens per tile).

Visual Tokens for a Tiled Image

$N_{\text{vis}} = (n_{\text{tiles}} + 1) \times 256$

The $+1$ is the thumbnail. The paper's $800 \times 1300$ example matches the 2:3 ratio ( $896 \times 1344$ ), giving $n_{\text{tiles}} = 6$ and $N_{\text{vis}} = 7 \times 256 = 1792$ visual tokens before any routing.

Key Points

1
Three training stages for InternVL3.5: native pre-training → SFT → Cascade RL. InternVL3.5-Flash adds ViCO on top
2
Pre-training updates all parameters jointly on ~116M samples / ~250B tokens, text-only : multimodal ≈ 1 : 2.5, max length 32K
3
Loss is computed only on text tokens (only response tokens for conversations); image tokens act purely as context
4
Square averaging weights each token by $1/N^{0.5}$ to avoid bias toward long or short responses
5
SFT: ~56M samples / ~130B tokens, text-only : multimodal ≈ 1 : 3.5, 32K context
6
SFT data = InternVL3 instruction data + filtered Thinking-mode reasoning traces + capability-expansion data (GUI, embodied, SVG)
7
The paper credits the large text-only share with preserving language skills: InternVL3.5-1B beats its own Qwen3-0.6B base on 15 of 16 text benchmarks (+6.7 overall)

Mathematical Formulation

Next-Token Prediction Loss

$\mathcal{L}_i = -\log p_\theta\!\left(x_i \mid x_1, \ldots, x_{i-1}\right)$

Computed on each text token $x_i$ . The prefix may contain both text and image tokens, so the image conditions the prediction but is never itself a prediction target.

Square-Averaged Loss Weighting

$\mathcal{L}_i' = \frac{w_i}{\sum_j w_j} \cdot \mathcal{L}_i, \qquad w_i = \frac{1}{N^{0.5}}$

$N$ is the number of loss-bearing tokens in the sample. Token averaging ( $w = 1$ ) favours long samples, and sample averaging ( $w = 1/N$ ) favours short ones. The square root splits the difference.

Key Points

1
RL's core advantage over SFT: negative samples prune low-quality regions of the model's output space
2
Offline RL (DPO family) is efficient and stable but has a lower ceiling. Online RL (PPO/GRPO/GSPO) reaches higher but is costly and needs good rollouts
3
Stage 1, MPO: $\mathcal{L}_{\text{MPO}} = w_p\mathcal{L}_p + w_q\mathcal{L}_q + w_g\mathcal{L}_g$ with DPO, BCO and LM losses, trained on MMPR-v1.2 (~200K pairs)
4
Stage 2, GSPO without a reference-model constraint: group-normalised advantages with a sequence-level importance ratio, trained on MMPR-Tiny (~70K queries with accuracy in 0.2–0.8)
5
Advantages claimed: stability (offline decoupling resists reward hacking, and a stronger start stabilises the online stage), efficiency (rollouts shared across models) and a higher ceiling
6
Table 16 (8B): Instruct 53.6 → MPO 56.3 (~0.3K h) | GSPO 1 ep 57.3 (~5.5K h) | GSPO 2 ep 58.2 (~11.0K h) | Cascade RL 60.3 (~5.8K h)
7
Cascade RL beats two full GSPO episodes using about half the GPU hours

Mathematical Formulation

MPO — The Offline Stage

$\mathcal{L}_{\text{MPO}} = w_p\,\mathcal{L}_p + w_q\,\mathcal{L}_q + w_g\,\mathcal{L}_g$

$\mathcal{L}_p$ is the DPO preference loss (chosen beats rejected), $\mathcal{L}_q$ the BCO quality loss (each response judged good or bad on its own), and $\mathcal{L}_g$ the ordinary LM loss on chosen responses, which keeps generation fluent.

Group-Normalised Advantage

$\hat{A}_i = \frac{r(x, y_i) - \operatorname{mean}\big(\{r(x, y_i)\}_{i=1}^{G}\big)}{\operatorname{std}\big(\{r(x, y_i)\}_{i=1}^{G}\big)}$

For each query $x$ , $G$ responses are sampled and each reward is standardised against its own group, as in GRPO. No value network is needed, because the group itself is the baseline.

GSPO Objective

$\mathcal{L}_{\text{GSPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, \{y_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(\cdot \mid x)}\left[\frac{1}{G}\sum_{i=1}^{G} \min\Big(s_i(\theta)\hat{A}_i,\; \operatorname{clip}\big(s_i(\theta), 1-\varepsilon, 1+\varepsilon\big)\hat{A}_i\Big)\right]$

A PPO-style clipped surrogate, except that clipping is applied once per response using the sequence-level ratio $s_i(\theta)$ , rather than once per token.

Sequence-Level Importance Ratio

$s_i(\theta) = \left(\frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_{\text{old}}}(y_i \mid x)}\right)^{\frac{1}{|y_i|}} = \exp\!\left(\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t})}\right)$

The geometric mean of the per-token ratios. Length normalisation keeps $s_i$ on the same scale for a 20-token and a 2,000-token answer, which (per the GSPO authors) avoids the high-variance token-level ratios that destabilise MoE training.

Key Points

1
InternVL3.5-Flash = InternVL3.5 + a second pixel shuffle ( $\xi = \tfrac{1}{16}$ , 64 tokens) + a per-patch router choosing between 64 and 256 tokens
2
Dynamic High Resolution adapts to image shape. ViR adds adaptivity to semantic content
3
ViCO stage 1, consistency training: the policy at random $\xi \in \{\tfrac{1}{4}, \tfrac{1}{16}\}$ matches a frozen reference that always uses $\xi = \tfrac{1}{4}$ , via KL
4
ViCO stage 2, router training: MLLM frozen, ViR trained as a binary classifier with cross-entropy
5
Router labels are self-generated from the loss ratio $r_i$ against a dynamic threshold $\tau$ (a $k$ -th percentile of a sliding window), which keeps targets balanced
6
ViCO uses ~30B tokens. Router data is mainly OCR and VQA, where resolution matters
7
Result: ~50% fewer visual tokens with near-identical scores, e.g. 8B overall 80.2 → 79.8, 235B MoE 85.0 → 84.5 (Table 17)

Mathematical Formulation

ViCO Consistency Loss

$\mathcal{L}_{\text{ViCO}} = \mathbb{E}_{\xi \sim \mathcal{R}}\left[\frac{1}{N}\sum_{i=1}^{N} \operatorname{KL}\Big(\pi_{\theta_{\text{ref}}}(y_i \mid y_{<i}, I) \,\Big\|\, \pi_{\theta_{\text{policy}}}(y_i \mid y_{<i}, I_\xi)\Big)\right]$

$\xi$ is drawn uniformly from $\{\tfrac{1}{4}, \tfrac{1}{16}\}$ , so $I_\xi$ is the image at 256 or 64 tokens per patch. The reference always sees the uncompressed $I$ (equivalently $\xi = \tfrac{1}{4}$ ). Minimising the KL makes the compressed view reproduce the full view's predictions token by token.

Per-Patch Loss Ratio

$r_i = \frac{\mathcal{L}_{\text{ViCO}}\big(y_i \mid I_{1/16}\big)}{\mathcal{L}_{\text{ViCO}}\big(y_i \mid I_{1/4}\big)}$

How much worse the model gets when this patch is squeezed to 64 tokens. A ratio near 1 means compression costs almost nothing.

Router Target

$y_i^{\text{router}} = \begin{cases} 0, & r_i < \tau \quad \text{(compress: } \xi = \tfrac{1}{16}\text{)} \\ 1, & r_i \ge \tau \quad \text{(keep: } \xi = \tfrac{1}{4}\text{)} \end{cases}$

$\tau$ is the $k$ -th percentile of recent $r_i$ values in a sliding window, so it tracks the current loss scale and keeps the two classes balanced.

Key Points

1
The ViT is parallel and stateless. The LLM is autoregressive and memory-bound. On shared GPUs each blocks the other
2
DvD places ViT + MLP (+ ViR) on a vision server and only the LLM on a language server
3
Communication is one-way BF16 features over TCP, optionally RDMA
4
Vision, transfer and language form an asynchronous three-stage pipeline, so ViT work overlaps LLM prefill and decode
5
Table 18, 38B: DvD gives 1.19x at 448px, 1.87x at 896px and 1.97x at 1344px. The speed-up grows with resolution
6
DvD + ViR: up to 4.05x (38B, 896px: 2.71 → 10.97 req/s) and 3.47x for 241B-A28B at the same resolution
7
Hardware for each side can be scaled independently, and new vision modules don't require redeploying the language server

Mathematical Formulation

Why Overlap Helps

$T_{\text{serial}} \approx \sum_{k}\big(t^{(k)}_{\text{vis}} + t^{(k)}_{\text{llm}}\big) \quad\longrightarrow\quad T_{\text{DvD}} \approx \sum_{k} \max\big(t^{(k)}_{\text{vis}},\, t^{(k)}_{\text{llm}}\big)$

A simplified pipeline view (not an equation from the paper) for a stream of requests $k$ . Once vision and language run concurrently on separate hardware, steady-state cost per request approaches the slower of the two stages rather than their sum. The larger $t_{\text{vis}}$ is (bigger ViT, more tiles), the more there is to hide, which matches the resolution trend in Table 18.

Key Points

1
Deep thinking: Thinking mode produces step-by-step reasoning with intermediate validation and adds depth
2
Parallel thinking: Best-of-N with VisualPRM-v1.1 as the critic adds breadth
3
Headline results in the paper are reported without TTS unless noted
4
TTS is applied only to reasoning benchmarks. Perception tasks showed no significant gain
5
Gains stack with Cascade RL: +2.6 (4B), +2.1 (8B), +1.8 (241B-A28B) points on overall reasoning
6
Small models benefit most: InternVL3.5-1B goes from 33.8 → 43.3 with parallel thinking (Table 3)

Mathematical Formulation

Best-of-N with a Process Reward Model

$\hat{y} = \arg\max_{y \in \{y_1, \ldots, y_N\}} \; \operatorname{PRM}(x, y), \qquad y_j \sim \pi_\theta(\cdot \mid x)$

Sample $N$ reasoning chains, have VisualPRM score each one (aggregating its per-step scores) and return the highest-scoring chain. Compute grows linearly with $N$ , and none of it is spent on training.

Key Points

1
General multimodal overall: 74.1 for InternVL3.5-241B-A28B vs 74.0 for GPT-5 (Table 2)
2
Reasoning: 77.7 MMMU, 82.7 MathVista, overall 67.1 (GPT-5 74.3, GLM-4.5V 68.4)
3
Text overall 85.3, ahead of every open MLLM in Table 2 (GPT-5 91.3)
4
Cascade RL lifts the reasoning average at every size, e.g. 2B 38.5 → 50.7 (+12.2) and 241B-A28B 60.4 → 66.9 (+6.5) over SFT (Table 15)
5
InternVL3.5-8B after SFT alone already beats InternVL3-8B by +9.3 (53.6 vs 44.3)
6
Mid-size MMMU: 30B-A3B 75.6 and 14B 73.3 both beat InternVL3-78B (72.2)
7
Language preserved: 1B beats Qwen3-0.6B on 15/16 text benchmarks (+6.7). 241B gains +2.3 over Qwen3-235B-A22B (Table 14)
8
Weak spot: HallusionBench 57.3 vs 65.4 (GLM-4.5V) and 65.2 (GPT-5)

Mathematical Formulation

Where the Reasoning Gain Comes From (241B-A28B)

$\underbrace{54.6}_{\text{InternVL3-78B}} \xrightarrow{\text{SFT}} \underbrace{60.4}_{\text{Instruct}} \xrightarrow{\text{MPO}} \underbrace{62.4}_{\text{MPO}} \xrightarrow{\text{GSPO}} \underbrace{66.9}_{\text{Cascade RL}}$

The averages over seven reasoning benchmarks in Table 15. Better data and SFT account for the first jump. Of the remaining 6.5 points from Cascade RL, the online GSPO stage contributes most, building on the MPO warm-up.

Key Points

1
ScreenSpot-v2 92.9 for 241B-A28B: +2.6 over UI-TARS-72B and close to Seed1.5-VL (95.2)
2
Online agents: WindowsAgentArena 18.0 (+8.3 over Qwen2.5-VL-72B, GPT-4o 3.5) and WebArena-Lite-v2 11.7 (GPT-4o 1.9)
3
VSI-Bench: 1B scores 49.3 (+19.6 over InternVL3-1B) and 241B-A28B scores 69.5
4
ERQA 46.8 vs Gemini-2.5-Pro 48.3. Highest embodied overall in Table 11 (55.8)
5
SGP-Bench 70.7 (best open model) vs GPT-5 77.5
6
Text2SVG FID 11.27 (241B). The 38B's 14.56 beats GPT-4o's 15.18
7
These skills come from the SFT capability-expansion data (GUI, embodied, SVG), not from architecture changes

Go deeper

Study Plans

VLM Study Plan Reinforcement Learning Study Plan All Papers Practice Problems

Paper Overview

Chapter Roadmap

The Open-Source Gap

ViT–MLP–LLM Family

Native Pre-Training + SFT

Cascade RL

ViR + ViCO

Test-Time Scaling

Decoupled Deployment

Results

GUI, Embodied, SVG

Concept by concept

The Open-Source Gap

Key Points

Further Reading

ViT–MLP–LLM and the Model Family

Key Points

Mathematical Formulation

Pixel Shuffle Token Budget

Visual Tokens for a Tiled Image

Further Reading

Native Pre-Training and SFT

Key Points

Mathematical Formulation

Next-Token Prediction Loss

Square-Averaged Loss Weighting

Further Reading

Cascade RL — Offline MPO, then Online GSPO

Key Points

Mathematical Formulation

MPO — The Offline Stage

Group-Normalised Advantage

GSPO Objective

Sequence-Level Importance Ratio

Further Reading

Visual Resolution Router and ViCO

Key Points

Mathematical Formulation

ViCO Consistency Loss

Per-Patch Loss Ratio

Router Target

Further Reading

Decoupled Vision-Language Deployment

Key Points

Mathematical Formulation

Why Overlap Helps

Further Reading

Test-Time Scaling — Deep and Parallel Thinking

Key Points

Mathematical Formulation

Best-of-N with a Process Reward Model

Further Reading

Results

Key Points

Mathematical Formulation

Where the Reasoning Gain Comes From (241B-A28B)

Further Reading

Beyond Benchmarks — GUI, Embodied and SVG

Key Points

Further Reading

Related Papers

Study Plans