Qwen3-VL
Interleaved MRoPE, DeepStack and Text Timestamps in an Open 2B–235B Vision-Language Family with Native 256K Context
Qwen Team
11
Concepts
10
Formulas
8
Figures
5
Demos
Paper Overview
Qwen3-VL (Qwen Team, Alibaba — November 2025) is the vision-language member of the Qwen3 family, and its report is built around a claim most earlier VLMs could not make: adding vision should not cost you language. The family spans four dense models (2B / 4B / 8B / 32B) and two mixture-of-experts models (30B-A3B and the 235B-A22B flagship), each trained to a native 256K-token context of interleaved text, images and video.
The report names three pillars: pure-text ability that matches or beats comparable text-only backbones, faithful retrieval across long documents and videos, and strong multimodal reasoning. Three architectural upgrades serve them. Interleaved MRoPE spreads the time, height and width axes across the whole rotary frequency spectrum instead of giving each a band of its own. DeepStack feeds features from three levels of the vision encoder into the first three LLM layers without adding a single token to the context. Text timestamps replace absolute-time position IDs with plain strings like <3.0 seconds>.
Around that sits a four-stage pre-training run (a merger-only warm-up, then roughly 1T tokens at 8K, roughly 1T at 32K and 100B at 262K), a square-root-normalised per-token loss that balances text against multimodal data, and a post-training pipeline that produces separate thinking and non-thinking variants through SFT, strong-to-weak distillation and RL.
Across its thinking and instruct variants the 235B-A22B flagship scores 80.6 on MMMU, 85.8 on MathVista-mini, 97.1 on DocVQA and 920 on OCRBench. It finds a needle frame in a 30-minute video with 100% accuracy, and still reaches 99.5% when extrapolated to a million tokens.
Chapter Roadmap
Click any topic to jump in
Vision Without Losing Language
A VLM is expected to match its text-only backbone on language benchmarks, not just add eyes to it.
Three Modules, Six Sizes
SigLIP-2 encoder, a 2x2 MLP merger and Qwen3 dense or MoE backbones from 2B to 235B-A22B.
Three architectural upgrades
Interleaved MRoPE
Time, height and width take turns across rotary frequencies instead of owning one band each.
DeepStack
Three ViT levels are added into LLM layers 1-3, with zero extra context length.
Text Timestamps
Strings like <3.0 seconds> replace absolute-time position IDs for video.
Recipe and data
Four Stages to 256K
Merger warm-up, then ~1T tokens at 8K, ~1T at 32K and 100B at 262K, with a square-root-normalised loss.
The Data Overhaul
Recaptioning, 39-language OCR, [0,1000] grounding coordinates, 3D boxes, STEM and agent data.
Alignment and agency
SFT, Distillation and RL
Thinking and non-thinking variants via SFT, strong-to-weak distillation and SAPO-based RL.
Thinking with Images
A think-act-observe visual agent trained with multi-turn, tool-integrated RL.
Evidence
Results
80.6 MMMU, 97.1 DocVQA, 920 OCRBench, and text scores that rival the text-only Qwen3 flagship.
A Needle in Two Hours of Video
100% needle retrieval up to 30 minutes (256K), 99.5% when extrapolated to 1M tokens.
Concept by concept
11 sections, in reading order.
Vision-language models have spread from captioning into long-document reading, STEM reasoning, GUI control and agent workflows. The report opens with the constraint that comes with this: these advances must not erode the underlying LLM's linguistic proficiency. Multimodal models are expected to match or surpass their text-only counterparts on language benchmarks. Historically they did not. Bolting a vision encoder onto an LLM and training on image-text data tended to wear away exactly the text skills that made the LLM worth using.
Qwen3-VL turns that expectation into a design target and organises the whole report around three pillars. The first is markedly stronger pure-text understanding, in several cases surpassing the text-only Qwen3 backbone it was built from. The second is robust long-context comprehension, with a native 256K-token window for text and interleaved multimodal input that supports retention, retrieval and cross-referencing across long documents and videos. The third is advanced multimodal reasoning across single images, multiple images and video.
The levers are spread across the stack. Architecture contributes three upgrades: interleaved MRoPE, DeepStack and text timestamps. The optimisation side moves from a per-sample loss to a square-root-normalised per-token loss that better balances text and multimodal data. The data side mixes text-only corpora into every pre-training stage. Post-training is split so that a thinking variant can spend tokens on reasoning while a non-thinking variant stays fast.
The evidence for the first pillar is the most interesting part of the paper, and it is mixed in an honest way. Against its own text-only sibling Qwen3-235B-A22B-Instruct-2507, the flagship Qwen3-VL-235B-A22B-Instruct scores higher on AIME-25 (74.7 vs 70.3), HMMT-25 (57.4 vs 55.4) and LiveCodeBench v6 (54.3 vs 51.8), but lower on MMLU-Pro (81.8 vs 83.0) and GPQA (74.3 vs 77.5). The cleaner wins are at mid-size, shown in the table here: Qwen3-VL-32B-Instruct beats the original Qwen3-32B on every text benchmark listed, for example 66.2 vs 20.2 on AIME-25 and 78.6 vs 71.9 on MMLU-Pro. That baseline is the original Qwen3-32B release, so newer post-training data probably explains part of the gap. Against the refreshed Qwen3-30B-A3B-Instruct-2507 the MoE Qwen3-VL-30B-A3B is roughly level (77.8 vs 78.4 MMLU-Pro) and ahead on AIME-25 (69.3 vs 61.3). Either way, the VLM is no longer the weaker text model.

Key Points
- 1
The stated bar: a multimodal model should match or surpass its text-only counterpart on language benchmarks, not merely tolerate a small regression
- 2
Three pillars: stronger pure-text understanding, a native 256K-token interleaved context, and advanced multimodal reasoning across images, multi-image and video
- 3
Six models: dense 2B / 4B / 8B / 32B and MoE 30B-A3B / 235B-A22B, all on Qwen3 backbones, each released in thinking and non-thinking (instruct) variants
- 4
Flagship vs its text-only sibling (Table 5): higher on AIME-25 (74.7 vs 70.3), HMMT-25 (57.4 vs 55.4) and LiveCodeBench v6 (54.3 vs 51.8); lower on MMLU-Pro (81.8 vs 83.0) and GPQA (74.3 vs 77.5)
- 5
Mid-size (Table 7): Qwen3-VL-32B-Instruct beats Qwen3-32B on all listed text benchmarks, e.g. AIME-25 66.2 vs 20.2, LiveCodeBench v6 43.8 vs 29.1
- 6
Text is protected by mixing text-only data into every pre-training stage and by a square-root-normalised per-token loss that keeps either modality from dominating
- 7
Released under Apache 2.0, which lets the whole family serve as an open base for agents and multimodal code tools
Qwen3-VL keeps the three-module layout of Qwen2.5-VL: a vision encoder, an MLP-based vision-language merger, and a large language model. The changes happen inside the modules and in how they connect, not in the overall shape.
The vision encoder is SigLIP-2, initialised from the official checkpoints and then trained further at dynamic input resolutions. To handle arbitrary image sizes it uses 2D-RoPE and interpolates its absolute position embeddings to the input size, following CoMP. The default is the SigLIP2-SO-400M variant; the 2B and 4B models use the smaller SigLIP2-Large (300M). The report's encoder ablation shows why the continued training is worth it. In the CLIP pre-training stage, the resulting Qwen3-ViT roughly matches SigLIP-2 on ImageNet variants but jumps from 36.9 to 45.5 on OmniBench, an in-house world-knowledge suite. Paired with the same 1.7B Qwen3 LLM and trained on 1.5T tokens, it is ahead on OCRBench, AI2D, RealWorldQA (58.7 to 66.1) and InfoVQA.
The merger is a two-layer MLP that compresses each 2x2 block of neighbouring encoder features into one visual token in the LLM's hidden dimension. Separate, specialised mergers serve DeepStack. The token counts annotated on Figure 1 reveal the net resolution of the system. A 1440x800 image becomes 1,125 tokens, a 1248x9376 page becomes 11,427 tokens, and a 256x32 strip becomes 8 tokens. Every one of these works out to exactly one visual token per 32x32-pixel area. Cost grows linearly with pixel count, so a tall document page costs ten times a photo.
The LLM is a Qwen3 backbone in all six sizes. The flagship Qwen3-VL-235B-A22B has 235B total parameters with 22B activated per token, so its per-token compute is closer to a 22B dense model than to its full size. Native-resolution input, multi-level injection and positional encoding all meet in Figure 1, which the next three sections take apart one piece at a time.

Key Points
- 1
Vision encoder: SigLIP-2, further trained at dynamic resolution with 2D-RoPE and interpolated absolute position embeddings (following CoMP)
- 2
SigLIP2-SO-400M by default; SigLIP2-Large (300M) for the small 2B and 4B models
- 3
Merger: a two-layer MLP compresses 2x2 encoder features into one visual token in the LLM's hidden size; extra mergers serve DeepStack
- 4
Figure 1's own numbers imply one token per 32x32 pixels: 1440x800 gives 1,125 tokens, 1248x9376 gives 11,427, and 256x32 gives 8
- 5
LLM: Qwen3 dense (2B / 4B / 8B / 32B) or MoE (30B-A3B, 235B-A22B); the flagship activates 22B of 235B parameters per token
- 6
Encoder ablation (Table 11): Qwen3-ViT vs SigLIP-2 lifts OmniBench 36.9 to 45.5 at the CLIP stage, and RealWorldQA 58.7 to 66.1 with a 1.7B Qwen3 LLM
Mathematical Formulation
Visual Token Count (implied by Figure 1)
Nvis=32H⋅32W,321440⋅32800=45⋅25=1125
Each LLM token covers a 32×32 pixel area once the 2×2 merge is applied, so token cost scales with pixel area. The 1248x9376 document page in Figure 1 gives 39⋅293=11,427 tokens, matching the figure's annotation exactly. The report does not print this formula; it is derived from the figure's numbers.
Rotary position embedding (RoPE) encodes position by rotating pairs of embedding dimensions. Pair j rotates at its own frequency θj. Low-index pairs spin fast and resolve fine, local position differences. High-index pairs spin slowly and stay informative over long distances. A position code is only as good as the spread of frequencies applied to it.
Qwen2-VL extended RoPE to images and video with MRoPE: each visual token gets a three-part position (temporal t, height h, width w), and the rotary dimensions are partitioned into three contiguous chunks, one per axis. Each chunk therefore owns a different slice of the frequency spectrum. One axis gets only the fast frequencies and another only the slow ones. The report calls this an imbalanced frequency spectrum, and notes that later studies showed it hurts long-video understanding. That is exactly where the temporal axis needs long-range resolution it may not have.
Qwen3-VL's fix, drawn from Huang et al. (2025), is to interleave the axes across the embedding dimensions instead of chunking them: t,h,w,t,h,w,…. Every axis now samples the full range from high to low frequency, so each one is represented uniformly across both bands. The demo below shows the difference directly. In chunked mode each axis covers about a third of the spectrum; interleaved, each covers nearly all of it.
The change costs nothing in parameters or compute. It is a re-assignment of which position component drives which dimension pair. The report credits it, together with text timestamps and denser video captions, with significantly better long-range positional modelling for video.
Key Points
- 1
RoPE rotates dimension pair j at frequency θj; fast pairs resolve local position, slow pairs carry long-range position
- 2
MRoPE (Qwen2-VL) gives each visual token a (t,h,w) position and splits the rotary dimensions into three contiguous chunks, one per axis
- 3
Chunking gives each axis a different frequency band, an imbalanced spectrum that later work showed degrades long-video benchmarks
- 4
Interleaved MRoPE cycles t,h,w across dimensions so each axis is uniformly represented in both low- and high-frequency bands
- 5
No new parameters and no extra compute: it only changes which position component drives which dimension pair
- 6
The report attributes improved long-range positional modelling for video to the change, alongside text timestamps
Mathematical Formulation
Rotary Frequencies
θj=b−2j/d,j=0,1,…,2d−1
Standard RoPE frequencies for a d-dimensional head with base b. Pair j is rotated by the angle p⋅θj for position p. Small j means high frequency and large j means low frequency.
Chunked Axis Assignment (Qwen2-VL)
axis(j)=⎩⎨⎧t,h,w,j<dtdt≤j<dt+dhotherwise
Each axis owns one contiguous block of pairs, so the temporal axis only ever sees the frequencies inside its own block.
Interleaved Axis Assignment (Qwen3-VL)
axis(j)=(t,h,w)jmod3,ϕj=paxis(j)⋅θj
The axes take turns, so each one is spread evenly from j=0 to j=d/2−1. The rotation angle ϕj uses the token's position along whichever axis owns pair j. These two assignments are the textbook way to read the report's description; the report itself does not print the formulas.
A standard VLM hands the LLM one thing: the last layer of the vision encoder, projected into token space and placed in the input sequence. Everything the encoder computed along the way, from edges and strokes in early layers to parts and layouts in the middle, is either compressed into that final representation or lost. For tasks that depend on fine visual detail, such as reading small text in an infographic, that bottleneck shows.
The original DeepStack (Meng et al., 2024) addressed this by stacking tokens from multi-scale visual inputs into successive LLM layers. Qwen3-VL changes the source: instead of multiple input scales, it taps intermediate layers of the ViT itself, selecting features from three distinct levels of the encoder. Each level gets its own dedicated merger, which projects it into visual tokens. Those tokens are then added directly to the hidden states of the first three LLM layers.
The word that matters is added. The extra visual information arrives through residual-style addition at positions that already exist, not by appending more tokens. So DeepStack enriches multi-level fusion without introducing extra context length, which matters a great deal in a model whose selling point is a 256K window filled with images and video.
The ablation (Table 12, shown here) uses an internal 15B-A2B LLM pre-trained on 200B tokens and evaluated with no post-training. DeepStack raises the 11-benchmark average from 74.7 to 76.0. The biggest gains are where fine detail matters: InfoVQA 71.9 to 74.2, OCRBench 81.0 to 83.6, DocVQA 89.5 to 91.1. It is not a uniform win, since TextVQA is flat (80.6 vs 80.5), but on balance it is a cheap and clear improvement.

Key Points
- 1
Baseline VLMs feed the LLM only the final ViT layer; detail encoded in intermediate layers has to survive compression into it
- 2
Original DeepStack stacked tokens from multi-scale inputs; Qwen3-VL instead takes features from three intermediate ViT levels
- 3
Each level passes through its own dedicated merger and is added to the hidden states of LLM layers 1, 2 and 3
- 4
Addition rather than concatenation means no extra context length: the 256K window holds the same number of visual tokens
- 5
Ablation (15B-A2B LLM, 200B tokens, no post-training): average 74.7 to 76.0, InfoVQA +2.3, OCRBench +2.6, DocVQA +1.6, MMMU +1.2
- 6
Gains concentrate on fine-grained document and text-in-image tasks; TextVQA is flat (80.6 vs 80.5)
Mathematical Formulation
DeepStack Injection
h(ℓ)←h(ℓ)+Mℓ(v(kℓ)),ℓ∈{1,2,3}
v(kℓ) is the ViT feature map at the selected encoder level kℓ. Mℓ is that level's dedicated merger, and h(ℓ) is the LLM hidden state at layer ℓ, updated at the visual-token positions. The sequence length of h is unchanged.
To understand when something happens in a video, a model needs a sense of time. Qwen2.5-VL provided it by tying the temporal position IDs of MRoPE to absolute time, a time-synchronised MRoPE. A frame 30 seconds in received a temporal ID proportional to 30 seconds, whatever the sampling rate.
The Qwen3-VL report identifies two problems with that design. First, for long videos the temporal IDs become excessively large and sparse, which degrades the model's ability to follow long temporal contexts. Second, learning the time-to-ID mapping requires training data uniformly distributed across many frame rates, which makes the training data expensive to build.
The replacement is almost disarmingly simple, adopted from TimeMarker (Chen et al., 2024): write the time down. Each video temporal patch is prefixed with a timestamp as an ordinary text string, such as <3.0 seconds>. During training the timestamps appear in both plain seconds and HMS (hours:minutes:seconds) formats, so the model learns to read either timecode representation.
The trade-off is explicit. A few extra text tokens per frame group bring a modest increase in context length. In return, time becomes something the model reads directly in its native modality, text, instead of something it must decode from rotary geometry. The report credits this with more precise temporal perception for time-aware tasks such as video grounding and dense captioning. The flagship instruct model reaches 64.8 mIoU on Charades-STA temporal grounding.
Key Points
- 1
Qwen2.5-VL used time-synchronised MRoPE: temporal position IDs tracked absolute time
- 2
Limitation 1: on long videos the temporal IDs become excessively large and sparse, hurting long temporal context
- 3
Limitation 2: learning the mapping needs training data spread uniformly across many frame rates, which is costly to build
- 4
Qwen3-VL prefixes each temporal patch with a text timestamp such as <3.0 seconds> (following TimeMarker)
- 5
Training mixes seconds and HMS timecode formats so the model can interpret diverse representations
- 6
Cost: a modest increase in context length. Benefit: more precise grounding and dense captioning, with 64.8 mIoU on Charades-STA for the flagship instruct model
Before the full model is assembled, the vision encoder itself is continually trained from SigLIP-2 at dynamic resolutions. Pre-training of the assembled model then runs in four stages that go from basic alignment to ultra-long context (Table 1).
S0, Vision-Language Alignment, trains only the MLP merger while both the vision encoder and the LLM stay frozen. About 67B tokens of high-quality image captions, visual knowledge and OCR at a sequence length of 8,192 bridge the modality gap cheaply before anything expensive moves. S1, Multimodal Pre-Training, unfreezes everything and trains end to end on about 1T tokens at 8,192. The mix pairs VL data (interleaved documents, grounding, VQA, STEM and a little video) with text-only data to keep the LLM's language strong. S2, Long-Context Pre-Training, quadruples the sequence length to 32,768 for another ~1T tokens. It raises the share of text-only data for long-form reading and adds much more video and agent instruction data. S3, Ultra-Long-Context Adaptation, pushes the window to 262,144 tokens on a focused 100B-token set weighted toward long video and long documents.
Running text-only and multimodal data through one model creates a balancing problem, and the report changes the loss to address it. A per-sample loss averages each sample's token losses, so a 50-token caption and a 30K-token document count equally, and the document's tokens are each down-weighted heavily. A per-token loss sums all tokens, so long samples dominate. Qwen3-VL uses a square-root-normalised per-token loss, which sits between the two. The report says this better balances the contributions of text and multimodal data, boosting multimodal performance without compromising text.
All of this runs on Alibaba Cloud's PAI-Lingjun with a Megatron-LM hybrid of tensor, pipeline, context, expert and ZeRO-1 data parallelism, sustaining high utilisation at scales of up to 10,000 GPUs.

Key Points
- 1
S0: train the merger only (ViT and LLM frozen) on ~67B tokens of captions, visual knowledge and OCR at 8,192
- 2
S1: unfreeze everything, ~1T tokens at 8,192, mixing VL data with text-only data to keep language ability
- 3
S2: sequence length x4 to 32,768, ~1T tokens, with more text-only long-form data plus more video and agent data
- 4
S3: 262,144-token sequences on a focused 100B-token set dominated by long-video and long-document tasks
- 5
The loss moves from per-sample to square-root-normalised per-token, balancing text and multimodal contributions
- 6
Infrastructure: Megatron-LM with TP + PP + CP + EP + ZeRO-1 DP, scaling to 10,000 GPUs
Mathematical Formulation
Three Ways to Weight a Sample of Length N
L=∑iw(Ni)∑t=1Niℓi,t,w(N)∈{N1,N1,1}
ℓi,t is the loss on token t of sample i. With w=1/N (per-sample) every sample counts equally. With w=1 (per-token) long samples dominate. With w=1/N, a sample's total weight grows as N, between the two. The report names square-root normalisation but does not print its exact form; this is the standard reading, shown for intuition.
The report spends more pages on data than on architecture, and the changes are broad. The common thread is to replace weak supervision with model-generated or model-verified supervision, and to cover capabilities that earlier Qwen-VL models only partly had.
Captions and interleaved documents. Web image-text pairs are recaptioned by a Qwen2.5-VL-32B fine-tuned for the job, which uses the original alt-text to write richer and more fluent captions. Deduplication runs on the recaptioned text, and clustering over visual embeddings finds sparse regions of the distribution for targeted augmentation. Interleaved web documents are domain-classified, with ads and clickbait filtered out. Book-scale documents are parsed by a fine-tuned Qwen2.5-VL-7B and merged into sequences of up to 256K tokens, with minimum page-count and image-to-text-ratio constraints.
OCR, documents and grounding. OCR now covers 39 languages, up from 10 (beyond Chinese and English) in Qwen2.5-VL. The data comes from ~30M in-house real-world samples labelled without human annotation and ~30M synthesised multilingual samples. Figure 2 shows the result: over 70% accuracy on 32 of the 39 languages. Document parsing draws on 3M Common Crawl PDFs across 10 document types plus 4M internal documents, annotated in a QwenVL-HTML format (element-level boxes) and a QwenVL-Markdown format (LaTeX tables). Grounding covers boxes, points and counting. Coordinates now use a normalised [0, 1000] system rather than Qwen2.5-VL's absolute pixels, for robustness to resolution and aspect ratio.
Space, code, STEM and agents. 3D grounding data pairs a single image with a referring expression and a 9-DoF 3D box, unified into one virtual camera following Omni3D. Spatial data teaches relations, affordances and action-conditioned queries. Code data spans text-only code from Qwen3-Coder and multimodal tasks such as screenshot-to-HTML, image-to-SVG and diagram-to-LaTeX. STEM data includes 6M programmatically rendered diagram captions, 60M+ K-12 and undergraduate exercises and 12M+ long chain-of-thought samples. Agent data covers GUI trajectories across desktop, mobile and web, multimodal function-calling, and image- and text-search trajectories.

Key Points
- 1
Recaptioning with a fine-tuned Qwen2.5-VL-32B; semantic dedup on the new text; embedding-space clustering to fill sparse regions
- 2
Interleaved books parsed by a fine-tuned Qwen2.5-VL-7B and merged into sequences of up to 256K tokens
- 3
OCR in 39 languages (from 10): ~30M in-house plus ~30M synthesised samples; over 70% accuracy on 32 of 39 languages
- 4
Documents: 3M Common Crawl PDFs (10 types x 300K) plus 4M internal documents, in QwenVL-HTML and QwenVL-Markdown formats
- 5
Grounding with boxes, points and counting, using normalised [0, 1000] coordinates; 3D grounding with 9-DoF boxes in a unified virtual camera
- 6
STEM: 1M point-grounding and 2M perception VQA on rendered diagrams, 6M diagram captions, 60M+ exercises, 12M+ long-CoT samples
- 7
Agents: cross-platform GUI trajectories, synthesised multimodal function-calling, and image and text search trajectories for long-tail entities
Mathematical Formulation
Normalised Grounding Coordinates
x^=1000⋅Wx,y^=1000⋅Hy,x^,y^∈[0,1000]
Boxes and points are expressed relative to image width W and height H instead of in absolute pixels. The same object then gets the same coordinates whatever resolution or aspect ratio the image is fed at, which also simplifies post-processing.
Further Reading
Post-training has three stages and two outputs. Each size is released as a non-thinking (instruct) model and a thinking model, and the data is split accordingly from the first step.
Supervised fine-tuning uses about 1.2M samples: one third text-only, two thirds image-text and video-text, multilingual, and including single- and multi-turn dialogues over single and multiple images. It runs as one epoch at 32K and a second at 256K, where long inputs such as hundred-page technical documents, whole textbooks and videos up to two hours are interleaved with 32K data. Query filtering (with Qwen2.5-VL) and response filtering (rules plus Qwen2.5-VL reward models) remove unverifiable, low-quality or language-mixed samples. Thinking models also get a long-CoT cold start set at roughly 1:1 vision-language to text. It applies one notable filter: any VL maths problem that Qwen3-30B-nothink solves without seeing the image is discarded, so the remaining problems genuinely need vision.
Strong-to-weak distillation follows Qwen3. In the off-policy phase the student learns from teacher outputs. In the on-policy phase the student generates responses and its logits are aligned to the teacher's by minimising KL divergence. It is done on text-only data, fine-tuning the LLM backbone, yet the report finds it improves reasoning on both text and multimodal tasks.
Reinforcement learning has two parts. Reasoning RL trains on tasks with deterministically checkable answers (maths, code, logic, grounding, visual puzzles). A preliminary 235B checkpoint samples 16 responses per query, and queries with no correct answer are dropped. After pruning low-value sources this leaves about 30K queries. Per model, queries with pass rate above 90% are filtered out, and training uses SAPO, a smooth adaptive policy-gradient method. General RL optimises instruction following and preference alignment with rule-based and model-based rewards (Qwen2.5-VL-72B or Qwen3 as judges). It also targets strong-but-wrong SFT priors with tasks such as counter-intuitive counting and clock reading, and uses a dedicated prompt set to penalise language mixing and repetition.
Key Points
- 1
SFT: ~1.2M samples (one third text-only), one epoch at 32K then one at 256K; separate plain and CoT formats for instruct vs thinking models
- 2
Long-CoT cold start: ~1:1 VL to text; keeps hard, long-answer problems; drops VL maths solvable without the image by Qwen3-30B-nothink
- 3
Strong-to-weak distillation: off-policy on teacher outputs, then on-policy KL alignment of student and teacher logits, using text-only data
- 4
Reasoning RL: ~30K verifiable queries, 16 rollouts each, filter queries with pass rate > 90%, optimise with SAPO
- 5
No explicit format reward (format is enforced by task prompts); a language-mismatch penalty curbs code-switching
- 6
General RL: instruction following + preference alignment, with rule-based and model-based rewards to limit reward hacking and false negatives
- 7
Targeted unlearning of strong-but-wrong SFT priors using counter-intuitive counting and complex clock reading tasks
Mathematical Formulation
On-Policy Distillation Objective
Ldistill=Ex,y∼πS(⋅∣x)∑tDKL(πT(⋅∣x,y<t)πS(⋅∣x,y<t))
The student πS generates its own response y, and at every position its next-token distribution is pulled toward the teacher πT. Sampling from the student is what makes it on-policy: the student is corrected on the states it actually visits. The report says only that student and teacher logits are aligned by minimising KL divergence. The forward direction shown here is one standard choice, not one the report states.
A model with a fixed-resolution view of an image can only reason about what it was shown. Thinking with images lets the model act on the image during reasoning, for example zooming into a region, then look at the result and keep going. Qwen3-VL learns this agent loop, think, act, analyse feedback, answer, through a two-stage bootstrapping process.
In the first stage the team synthesises a small cold-start set of about 10K grounding examples, mostly simple two-turn VQA tasks such as attribute detection. They fine-tune Qwen2.5-VL-32B on it to imitate a visual agent, then strengthen it with multi-turn, tool-integrated RL. In the second stage that trained agent is distilled into a larger and more varied set of about 120K multi-turn agentic interactions. The same cold-start SFT plus tool-integrated RL pipeline is then applied to Qwen3-VL. An older model is used to manufacture the training trajectories for the new one.
RL uses three rewards. An answer-accuracy reward (judged by Qwen3-32B) checks the final answer. A multi-turn reasoning reward (judged by Qwen2.5-VL-72B) checks that the agent correctly interpreted tool feedback and reasoned coherently. A tool-calling reward compares the number of tool calls against an expert target that Qwen2.5-VL-72B estimates offline from task complexity. The third reward exists because of a failure the team observed: early models degenerated into making a single tool call to collect the first two rewards whatever the task required.
The payoff appears in fine-grained perception. With tools, the flagship instruct model reaches 93.7 on V∗ against 85.9 for the thinking model without tools. The 32B goes from 84.8 to 91.1 and the 8B from 77.5 to 90.1. The report's conclusion is that the gains from adding tools consistently exceed the gains from scaling model size, which it takes as evidence that tool-integrated agentic learning is a promising direction.
Key Points
- 1
The agent loop: think, act (e.g. zoom or crop), analyse feedback, answer, trained as multi-turn tool-integrated RL
- 2
Stage 1: ~10K cold-start grounding examples, SFT plus RL on Qwen2.5-VL-32B to build a visual agent
- 3
Stage 2: that agent is distilled into ~120K multi-turn interactions, and the same SFT + RL pipeline is applied to Qwen3-VL
- 4
Three rewards: answer accuracy (Qwen3-32B judge), multi-turn reasoning (Qwen2.5-VL-72B judge), and tool-calling against an expert-estimated target count
- 5
The tool-call reward fixes observed reward hacking: models collapsing to a single tool call regardless of task difficulty
- 6
With tools: V∗ 93.7 for the flagship instruct (vs 85.9 thinking without tools), 91.1 for the 32B, 90.1 for the 8B
- 7
Table 2 lists HRBench-4K / 8K at 85.4 / 82.4 with tools; the prose says 85.3 / 82.3, a small internal inconsistency
The flagship comparison (Table 2) puts Qwen3-VL-235B-A22B, in thinking and instruct forms, against Gemini 2.5 Pro, GPT-5 and Claude Opus 4.1 across roughly fifty benchmarks. The fair summary is that Qwen3-VL leads clearly in some areas, is competitive in most, and trails in a few.
Where it leads. Documents and OCR are the strongest category: 97.1 DocVQA, 920 OCRBench and 89.2 InfoVQA for the instruct model, plus the best OmniDocBench edit distances in the table (0.143 English). Visual maths is close behind, with 85.8 MathVista-mini and 74.6 MathVision for the thinking model, both best in the table. Grounding and embodied tasks are where the gap is widest: 92.1 RefCOCO average, 48.6 ODinW-13, 69.9 RefSpatialBench against 36.5 for Gemini 2.5 Pro. On HallusionBench the thinking model's 66.7 beats Gemini 2.5 Pro, GPT-5 and Claude Opus 4.1 by 3.0, 1.0 and 6.3 points. Among GUI agents it scores 62.0 on ScreenSpot Pro and 63.7 on AndroidWorld.
Where it trails. On MMMU, GPT-5 high (84.2) and Gemini 2.5 Pro thinking (81.7) beat Qwen3-VL's 80.6. Gemini 2.5 Pro leads general VQA overall, and Gemini 2.5 Pro or GPT-5 tops most of the video benchmarks (Video-MME: 85.1 for Gemini vs 79.2). On OSWorld, Claude Opus 4.1's 44.4 is ahead of Qwen3-VL's 38.1. The report is also candid that the video comparison cannot be fully fair. Qwen3-VL was allowed up to 2,048 frames, while API limits capped Gemini at 512, GPT-5 at 256 and Claude at 100.
Text and scale. On text-only benchmarks (Table 5) the VLM flagship is competitive with its text-only sibling and beats DeepSeek V3 0324 and Claude Opus 4 (non-thinking) on maths and coding. Down the size range, the 32B and 30B-A3B mostly beat Gemini 2.5 Flash and GPT-5 mini, and the 8B is reported competitive with the previous generation's Qwen2.5-VL-72B on video.


Key Points
- 1
Documents and OCR: DocVQA 97.1, OCRBench 920, InfoVQA 89.2, CC-OCR 82.2 (instruct), the strongest category against frontier models
- 2
Visual maths: MathVista-mini 85.8, MathVision 74.6, MathVerse-mini 85.0 (thinking), each best in Table 2
- 3
Grounding and spatial: RefCOCO-avg 92.1, ODinW-13 48.6 mAP, RefSpatialBench 69.9 (vs 36.5 Gemini 2.5 Pro), EmbSpatialBench 84.3
- 4
Alignment: HallusionBench 66.7, beating Gemini 2.5 Pro / GPT-5 / Claude Opus 4.1 by 3.0 / 1.0 / 6.3 points; MIA-Bench 92.7
- 5
Trails on: MMMU (80.6 vs GPT-5 high 84.2), Video-MME (79.2 vs Gemini 85.1), OSWorld (38.1 vs Claude 44.4)
- 6
Fairness caveat: video evaluation allowed Qwen3-VL 2,048 frames vs 512 / 256 / 100 for Gemini / GPT-5 / Claude due to API limits
- 7
Text (Table 5): AIME-25 74.7, LiveCodeBench v6 54.3, ahead of DeepSeek V3 0324 (46.6 / 45.2) and Claude Opus 4 non-thinking (33.9 / 44.6)
- 8
Agents: ScreenSpot Pro 62.0, OSWorldG 68.3, AndroidWorld 63.7; the 32B reaches 41.0 on OSWorld
Mathematical Formulation
HallusionBench Margins (Thinking Models)
vs Gemini 2.5 Pro:66.7−63.7vs GPT-5:66.7−65.7vs Claude Opus 4.1:66.7−60.4=3.0=1.0=6.3
Qwen3-VL-235B-A22B-Thinking against Gemini 2.5 Pro, GPT-5 and Claude Opus 4.1 thinking modes. These are the margins quoted in the report's text, recomputed from Table 2.
A 256K-token window is only useful if the model can find things in it. The report tests this directly with a video needle-in-a-haystack evaluation of Qwen3-VL-235B-A22B-Instruct. A semantically salient needle frame containing the critical visual evidence is inserted at a chosen depth in a long video. The model must locate the frame and answer a question about it.
Videos are sampled at 1 FPS, and frame resolution is adjusted dynamically to keep a constant visual token budget. Figure 3 sweeps video length and needle depth. Inside the training context, up to 30 minutes or 256K tokens, accuracy is a perfect 100% at every depth. Beyond it, the context is extended with YaRN-based positional scaling to as many as 1M tokens, about two hours of video, a 4x extrapolation past the trained window, and accuracy holds at 99.5%.
Two caveats keep this in perspective. A needle test measures retrieval, not reasoning, since finding one salient frame is easier than tracking a plot across an hour. And keeping the token budget constant means long videos are seen at lower resolution per frame. The document-side numbers are the more demanding test. On MMLongBench-Doc, multi-page PDF question answering, the flagship reaches 57.0% / 56.2% (instruct / thinking), which the report calls state of the art among the flagship models compared.
The video evaluation setup elsewhere in the paper shows how far the window is used. Benchmarks cap input at 2,048 frames with at most 224K video tokens, using 640 tokens per frame (768 for VideoMMMU and MMVU), sampled at 2 FPS (4 FPS for Charades-STA).

Key Points
- 1
Video NIAH: a salient needle frame is inserted at varying depths, and the model must locate it and answer a question about it
- 2
Sampling at 1 FPS with dynamic frame resolution to hold the visual token budget constant
- 3
100% accuracy at all depths for videos up to 30 minutes (256K tokens)
- 4
With YaRN extension to 1M tokens (~2 hours), a 4x extrapolation beyond training, accuracy stays at 99.5%
- 5
Long documents: 57.0 / 56.2 on MMLongBench-Doc (instruct / thinking), best among the compared flagships
- 6
Video benchmarks run with up to 2,048 frames and 224K video tokens, 640 tokens per frame (768 for VideoMMMU, MMVU)
- 7
Caveat: needle retrieval tests finding, not long-range reasoning, and constant budgets mean lower per-frame resolution on longer videos
Mathematical Formulation
Extrapolation Ratio
256K tokens1M tokens≈4×,accuracy: 100%→99.5%
Going four times past the trained context with YaRN positional scaling costs half a percentage point of needle-retrieval accuracy on Qwen3-VL-235B-A22B-Instruct.
Keep reading
Related Papers
Go deeper