Qwen2-VL

Qwen2-VL keeps the Qwen-VL layout: a Vision Transformer feeds a language model. The language side is upgraded to the Qwen2 LLM series. The vision side is a single ViT of about 675M parameters, initialised from the DFN ViT but with its fixed absolute position embedding replaced by 2D-RoPE.

The notable design choice is that the ViT does not grow with the model. All three sizes share the same 675M encoder, paired with a 1.5B, 7.6B or 72B LLM. As the paper puts it, this keeps the computational load of the ViT constant regardless of LLM scale. Vision cost therefore depends on how many pixels you send, not on which model you chose. That is the property Naive Dynamic Resolution builds on.

Between the two sits a deliberately simple connector: an MLP that merges each 2×2 neighbourhood of ViT tokens into one token, with <|vision_start|> and <|vision_end|> marking the visual span. There is no Q-Former and no fixed query budget. The number of visual tokens is set by the image, not by the architecture.

The core change sounds almost too simple: stop resizing. Qwen2-VL feeds each image to the ViT at (close to) its native resolution and lets the token count follow the pixel count. The idea, credited to NaViT (Patch n' Pack), had already shipped in the internal Qwen-VL Plus and Max models; Qwen2-VL brings it to the open release.

Two things have to change for this to work. First, a ViT with learned absolute position embeddings has a fixed grid baked in, so the paper removes them and uses 2D-RoPE. Rotary position is a function of relative offsets in height and width, so it is defined for any grid size. Second, the token count can get large, so after the ViT (patch size 14) an MLP merges each 2×2 block of tokens into one. The net rate is one LLM token per $28 \times 28$ pixel block. That gives the paper's worked example: a $224 \times 224$ image becomes $8 \times 8 = 64$ tokens, plus <|vision_start|> and <|vision_end|>, for 66 tokens in total. The images in Figure 2 show the range: a 224×28 formula strip costs 8 tokens, a 1260×700 photo costs 1,125, and a 1092×8204 web page costs 11,427.

Variable-length images are awkward to batch, so at inference images of different resolutions are packed into a single sequence, with the packed length capped to bound GPU memory. For evaluation the paper bounds the token count per image with min_pixels $= 100 \times 28 \times 28$ and max_pixels $= 16384 \times 28 \times 28$, i.e. between 100 and 16,384 tokens.

The ablation (Table 7) shows why this beats picking one good fixed size. Qwen2-VL-7B with a fixed 64 tokens collapses on InfoVQA (28.85). At 3,136 fixed tokens it does best on InfoVQA (77.27) but falls back on OCRBench (786). Dynamic resolution averages 1,924 tokens and is near the top everywhere, with 866 on OCRBench, the best of any setting. No single fixed resolution is optimal across benchmarks. Figure 4 adds a caveat: raising min_pixels to upscale small images helps perceptual tasks, but too high a value hurts OCRBench, whose many tiny images become out-of-distribution when blown up.

Once images arrive at their own resolution, the LLM also needs to know where each visual token came from. Standard 1D RoPE gives every token one integer position. Flattening an image row by row makes vertically adjacent patches look a full row width apart. For video, frame boundaries disappear into one long count.

M-RoPE decomposes the rotary embedding into three components: temporal, height and width. Each token gets a position triple $(t, h, w)$, and each component drives its own share of the rotary dimensions. The assignment rules are simple:

• Text: all three components use the same ID, so M-RoPE is functionally identical to 1D RoPE. A language-only prompt behaves exactly as it did in Qwen2.
• Images: the temporal ID is constant across the image; height and width IDs follow the token's row and column.
• Video: the temporal ID increments per frame, and height/width follow the same pattern as images.
• Mixed input: each modality's numbering starts at the previous modality's maximum ID plus one.

In Figure 3, the video's grid IDs reach at most 3, so the caption 'This video features…' resumes at $(4,4,4)$. That illustrates a less obvious benefit the paper highlights: M-RoPE reduces the position ID values for images and video. A 1,000-token image does not consume 1,000 positions, only about as many as its longest side. Smaller IDs are what let the model extrapolate to longer sequences at inference.

The evidence comes in two parts. In a controlled pre-training ablation (Qwen2-1.5B + ViT-L, Table 8), M-RoPE beats 1D-RoPE on most benchmarks, with the largest gains on video: NextQA 43.9 → 46.0, STAR 55.5 → 57.9, PerceptionTest 46.6 → 47.4, and MathVista 39.2 → 43.4. It is not uniform; RealWorldQA (54.5 → 53.7) and InfoVQA (50.8 → 50.3) dip slightly. On length extrapolation (Figure 5), Qwen2-VL-72B was trained with at most 16,384 tokens per video, yet holds its Video-MME accuracy at inference lengths up to 80K tokens.

Qwen2-VL trains on a mix of image and video data and runs both through one pipeline, instead of keeping a separate video branch. Three decisions make that work.

Sample densely. Each video is sampled at two frames per second, to keep as much temporal information as possible. Encode tubes, not patches. The ViT front-end uses 3D convolutions with a depth of two, following ViViT-style tubelet embedding. Each token covers two consecutive frames, which handles more frames without increasing sequence length. Make images a degenerate video. For consistency, every image is treated as two identical frames, so the same 3D convolution serves both modalities and there is no image-only path.

Long videos then run into the token budget. Rather than dropping frames, Qwen2-VL dynamically lowers the resolution of each frame so that the total tokens per video stay at or below 16,384. This is dynamic resolution in the temporal direction: a short clip gets sharp frames, a long one gets more but coarser frames. Combined with M-RoPE's per-frame temporal IDs, this is what allows the paper's claim of understanding videos over 20 minutes long.

There is a cost the paper acknowledges. At evaluation, frames per video were capped at 768, which the authors say may have hurt Video-MME on hour-long videos.

Training follows the Qwen-VL template in three stages, and the order of what gets unfrozen matters.

Stage 1 trains only the ViT, on a large corpus of image-text pairs, plus OCR and image classification data, for about 600B tokens. The LLM starts from Qwen2 and the ViT from DFN (with RoPE-2D replacing its fixed position embedding). The goal is to teach the new native-resolution encoder to produce features the LLM can use, without disturbing the LLM.

Stage 2 unfreezes everything and trains on 800B more tokens of image-related data. This includes more interleaved image-text content, VQA, multitask data, and continued pure-text data to keep the language ability intact. Across both stages the model sees 1.4T tokens, and image tokens are input only: supervision is applied to text tokens alone. The data mixes cleaned web pages, open-source datasets and synthetic data, with a knowledge cutoff of June 2023.

Stage 3 locks the ViT and instruction-tunes the LLM on ChatML-formatted data. This covers pure-text dialogue plus image QA, document parsing, multi-image comparison, video comprehension, video-stream dialogue and agent interactions.

The data format uses special tokens. <|vision_start|>/<|vision_end|> wrap visual content. For grounding, boxes are written as text coordinates normalised to [0, 1000) between <|box_start|> and <|box_end|>, and the referred phrase sits between <|object_ref_start|> and <|object_ref_end|>. Agent tasks are posed as multi-step decision problems: the system prompt lists permissible actions, and the model emits *FUNCTION*/*ARGS* calls with screenshots returned as *RESULT*.

Infrastructure is described too. Training ran on Alibaba Cloud PAI-Lingjun with 3D parallelism (DP + TP + PP), ZeRO-1, sequence parallelism with selective activation checkpointing, 1F1B pipeline parallelism for the 72B model, PyTorch 2.1.2 / CUDA 11.8, and flash-attention in both the ViT and the LLM.

The headline comparison (Table 2) puts Qwen2-VL-72B against the previous state of the art, Claude 3.5 Sonnet and GPT-4o. The pattern matches the paper's thesis: native resolution helps most where detail matters.

On documents and OCR it leads. It scores 96.5 on DocVQA (GPT-4o 92.8, Claude 3.5 Sonnet 95.2), 84.5 on InfoVQA (previous best 82.0), and 877 on OCRBench (previous best 852, GPT-4o 736). On general VQA it reaches 77.8 on RealWorldQA vs GPT-4o's 75.4, 74.0 on MMVet, 71.7 on MMT-Bench vs a previous best of 63.4, and 2482.7 on MME. On multilingual OCR, it scores 30.9 on MTVQA, and on the internal benchmark (Table 3) it beats GPT-4o in every language except Arabic (70.7 vs 75.9). MathVista reaches 70.5. The smaller models hold up: the 7B scores 94.5 on DocVQA and 866 on OCRBench, and the 2B scores 90.1 on DocVQA.

Video (Table 4) is strong at 72B: 73.6 MVBench, 68.0 PerceptionTest and 77.9 EgoSchema, ahead of GPT-4o's 72.2. On Video-MME it scores 71.2 / 77.8 (without / with subtitles), roughly level with GPT-4o and below Gemini 1.5-Pro's 75.0 / 81.3. The authors attribute part of this to the 768-frame evaluation cap.

The paper is explicit about the gaps. On MMMU (64.5 vs GPT-4o's 69.1), MMMU-Pro (46.2 vs 51.9) and MathVision (25.9 vs 30.4), Qwen2-VL-72B trails. These are the college-level, reasoning-heavy benchmarks, and the authors read them as the model's room to improve on complex problems. Grounding (Table 6) is strong for a generalist: 93.2 on RefCOCO val, up from Qwen-VL's 89.4.

The paper positions Qwen2-VL as a general vision-language agent. Every task, whether phone UI, robotics, card games or navigation, is framed as sequential decision-making. The model observes a screenshot, reasons, picks an action from a defined set, receives the next observation, and repeats. The (observation, action) history stays in a 32K context, with a new image appended after every action.

Function calling comes first, on an internal benchmark built because no public LVLM function-calling benchmark existed. Qwen2-VL-72B beats GPT-4o on both picking the right function (Type Match 93.1 vs 90.2) and passing exactly the right arguments read from the image (Exact Match 53.2 vs 50.0). The authors attribute GPT-4o's deficit to conservative tool avoidance and weaker OCR, especially on Chinese text.

UI operation shows the largest margin. On AITZ, Qwen2-VL reaches 89.6 TM / 72.1 EM against a previous best of 83.0 / 47.7 and GPT-4o's 70.0 / 35.3. This is where native-resolution grounding pays off: exact match requires tapping the right coordinates. Card games from RL4VLM need OCR as well as planning; Number Line and EZPoint reach 100.0 success. ALFRED robotic control (with SAM for object localisation) edges past the specialist ThinkBot, at 67.8 SR / 75.8 GC.

Navigation is the clear failure. On R2R, Qwen2-VL scores 51.7 against a specialist's 79.0, and on REVERIE 31.0 against 61.0. It is roughly level with GPT-4o, and both are far behind specialist models. The authors blame incomplete, unstructured map information assembled from multiple images, and call 3D map and location modelling a major open challenge for multimodal models.

The paper also makes a first pass at scaling behaviour for LVLMs, along two axes (Figure 6).

Model size (Figure 6a): capabilities are grouped into college-level problem solving (MMMU), math (MathVista + MathVision), document and table understanding, general VQA, and video. Performance rises consistently from 2B to 7B to 72B. Mathematical ability correlates most clearly with parameter count, while OCR-heavy tasks are already strong at small scale. This fits the architecture: perception lives largely in the shared 675M ViT, while multi-step reasoning lives in the LLM.

Training data (Figure 6b): during stage-2 pre-training of Qwen2-VL-7B, performance generally improves with more tokens. General VQA fluctuates, while text-and-graphics tasks such as AI2D and InfoVQA improve steadily.

The limits are stated plainly across the paper. Qwen2-VL-72B still lags GPT-4o on MMMU, the hardest reasoning benchmark. Bigger is not always better for inputs: upscaling small images past a point pushes them out of distribution and hurts OCRBench. Long video is bounded by the 768-frame evaluation cap, and the authors list longer sequences as future work. Embodied navigation remains far behind specialist models. Qwen2.5-VL, the successor, addresses several of these directly, with absolute-time MRoPE, window attention in the ViT and stronger agent grounding.