v9.1.1
Menu
Paper breakdownVision-LanguageFREE2025

Qwen3-VL

Interleaved MRoPE, DeepStack and Text Timestamps in an Open 2B–235B Vision-Language Family with Native 256K Context

Qwen Team

11

Concepts

10

Formulas

8

Figures

5

Demos

Paper Overview

Qwen3-VL (Qwen Team, Alibaba — November 2025) is the vision-language member of the Qwen3 family, and its report is built around a claim most earlier VLMs could not make: adding vision should not cost you language. The family spans four dense models (2B / 4B / 8B / 32B) and two mixture-of-experts models (30B-A3B and the 235B-A22B flagship), each trained to a native 256K-token context of interleaved text, images and video.

The report names three pillars: pure-text ability that matches or beats comparable text-only backbones, faithful retrieval across long documents and videos, and strong multimodal reasoning. Three architectural upgrades serve them. Interleaved MRoPE spreads the time, height and width axes across the whole rotary frequency spectrum instead of giving each a band of its own. DeepStack feeds features from three levels of the vision encoder into the first three LLM layers without adding a single token to the context. Text timestamps replace absolute-time position IDs with plain strings like <3.0 seconds>.

Around that sits a four-stage pre-training run (a merger-only warm-up, then roughly 1T tokens at 8K, roughly 1T at 32K and 100B at 262K), a square-root-normalised per-token loss that balances text against multimodal data, and a post-training pipeline that produces separate thinking and non-thinking variants through SFT, strong-to-weak distillation and RL.

Across its thinking and instruct variants the 235B-A22B flagship scores 80.6 on MMMU, 85.8 on MathVista-mini, 97.1 on DocVQA and 920 on OCRBench. It finds a needle frame in a 30-minute video with 100% accuracy, and still reaches 99.5% when extrapolated to a million tokens.

Chapter Roadmap

Click any topic to jump in

Interactive demo
1
Vision Without Losing Language

A VLM is expected to match its text-only backbone on language benchmarks, not just add eyes to it.

built as
2
Three Modules, Six Sizes

SigLIP-2 encoder, a 2x2 MLP merger and Qwen3 dense or MoE backbones from 2B to 235B-A22B.

upgraded with

Three architectural upgrades

3
Interleaved MRoPE

Time, height and width take turns across rotary frequencies instead of owning one band each.

4
DeepStack

Three ViT levels are added into LLM layers 1-3, with zero extra context length.

5
Text Timestamps

Strings like <3.0 seconds> replace absolute-time position IDs for video.

pre-trained via

Recipe and data

6
Four Stages to 256K

Merger warm-up, then ~1T tokens at 8K, ~1T at 32K and 100B at 262K, with a square-root-normalised loss.

7
The Data Overhaul

Recaptioning, 39-language OCR, [0,1000] grounding coordinates, 3D boxes, STEM and agent data.

then post-trained via

Alignment and agency

8
SFT, Distillation and RL

Thinking and non-thinking variants via SFT, strong-to-weak distillation and SAPO-based RL.

9
Thinking with Images

A think-act-observe visual agent trained with multi-turn, tool-integrated RL.

evaluated by

Evidence

10
Results

80.6 MMMU, 97.1 DocVQA, 920 OCRBench, and text scores that rival the text-only Qwen3 flagship.

11
A Needle in Two Hours of Video

100% needle retrieval up to 30 minutes (256K), 99.5% when extrapolated to 1M tokens.

Concept by concept

11 sections, in reading order.

01 / 11Vision Without Losing Language

Vision-language models have spread from captioning into long-document reading, STEM reasoning, GUI control and agent workflows. The report opens with the constraint that comes with this: these advances must not erode the underlying LLM's linguistic proficiency. Multimodal models are expected to match or surpass their text-only counterparts on language benchmarks. Historically they did not. Bolting a vision encoder onto an LLM and training on image-text data tended to wear away exactly the text skills that made the LLM worth using.

Qwen3-VL turns that expectation into a design target and organises the whole report around three pillars. The first is markedly stronger pure-text understanding, in several cases surpassing the text-only Qwen3 backbone it was built from. The second is robust long-context comprehension, with a native 256K-token window for text and interleaved multimodal input that supports retention, retrieval and cross-referencing across long documents and videos. The third is advanced multimodal reasoning across single images, multiple images and video.

The levers are spread across the stack. Architecture contributes three upgrades: interleaved MRoPE, DeepStack and text timestamps. The optimisation side moves from a per-sample loss to a square-root-normalised per-token loss that better balances text and multimodal data. The data side mixes text-only corpora into every pre-training stage. Post-training is split so that a thinking variant can spend tokens on reasoning while a non-thinking variant stays fast.

The evidence for the first pillar is the most interesting part of the paper, and it is mixed in an honest way. Against its own text-only sibling Qwen3-235B-A22B-Instruct-2507, the flagship Qwen3-VL-235B-A22B-Instruct scores higher on AIME-25 (74.7 vs 70.3), HMMT-25 (57.4 vs 55.4) and LiveCodeBench v6 (54.3 vs 51.8), but lower on MMLU-Pro (81.8 vs 83.0) and GPQA (74.3 vs 77.5). The cleaner wins are at mid-size, shown in the table here: Qwen3-VL-32B-Instruct beats the original Qwen3-32B on every text benchmark listed, for example 66.2 vs 20.2 on AIME-25 and 78.6 vs 71.9 on MMLU-Pro. That baseline is the original Qwen3-32B release, so newer post-training data probably explains part of the gap. Against the refreshed Qwen3-30B-A3B-Instruct-2507 the MoE Qwen3-VL-30B-A3B is roughly level (77.8 vs 78.4 MMLU-Pro) and ahead on AIME-25 (69.3 vs 61.3). Either way, the VLM is no longer the weaker text model.

Vision Without Losing Language diagram

Key Points

  • 1

    The stated bar: a multimodal model should match or surpass its text-only counterpart on language benchmarks, not merely tolerate a small regression

  • 2

    Three pillars: stronger pure-text understanding, a native 256K-token interleaved context, and advanced multimodal reasoning across images, multi-image and video

  • 3

    Six models: dense 2B / 4B / 8B / 32B and MoE 30B-A3B / 235B-A22B, all on Qwen3 backbones, each released in thinking and non-thinking (instruct) variants

  • 4

    Flagship vs its text-only sibling (Table 5): higher on AIME-25 (74.7 vs 70.3), HMMT-25 (57.4 vs 55.4) and LiveCodeBench v6 (54.3 vs 51.8); lower on MMLU-Pro (81.8 vs 83.0) and GPQA (74.3 vs 77.5)

  • 5

    Mid-size (Table 7): Qwen3-VL-32B-Instruct beats Qwen3-32B on all listed text benchmarks, e.g. AIME-25 66.2 vs 20.2, LiveCodeBench v6 43.8 vs 29.1

  • 6

    Text is protected by mixing text-only data into every pre-training stage and by a square-root-normalised per-token loss that keeps either modality from dominating

  • 7

    Released under Apache 2.0, which lets the whole family serve as an open base for agents and multimodal code tools