Qwen2.5

The open-weight release is seven dense decoder-only models — 0.5B, 1.5B, 3B, 7B, 14B, 32B and 72B — each as a base and an instruction-tuned model, plus quantized variants; over 100 checkpoints in total. Qwen2 had skipped 3B, 14B and 32B. Qwen2.5 brings them back on the argument that these middle sizes are the most cost-effective for resource-limited deployments and are under-served by other open families.

The architecture is deliberately conservative and identical to Qwen2: a pre-norm Transformer decoder with RMSNorm, SwiGLU feed-forward layers, RoPE positions, QKV bias in attention, and Grouped Query Attention. Table 1 shows the GQA ratios. The 72B model has 64 query heads but only 8 key-value heads, so each KV head is shared by 8 query heads and the KV cache is 8x smaller than full multi-head attention would need. The three smallest models also tie input and output embeddings, which matters when a 151K-token vocabulary is a large fraction of a 0.5B model.

Two proprietary models, Qwen2.5-Turbo and Qwen2.5-Plus, are Mixture-of-Experts variants served via API. They replace each FFN with an MoE layer that routes every token to its top-$K$ experts, using the fine-grained expert segmentation and shared-expert routing of Qwen1.5-MoE and DeepSeekMoE: many small experts rather than a few large ones, plus experts every token always passes through. The report does not publish their expert counts or parameter sizes; it says only that hyper-parameters were tuned so they match specific dense models — Plus is positioned against Qwen2.5-72B, Turbo against Qwen2.5-14B.

The tokenizer is unchanged in substance — byte-level BPE with 151,643 regular tokens — but the control-token set grows from 3 to 22, including two new tokens for tool use. Every Qwen2.5 model shares this vocabulary, so tool-calling formats and chat templates transfer across the whole family.

Scaling laws are usually used to answer one question: given a compute budget, how big should the model be and how many tokens should it see? Chinchilla and Llama 3 use them that way. Qwen2.5 turns them toward a more operational question: given a model of size $N$ trained on $D$ tokens, what learning rate and batch size should it use?

This matters because Qwen2.5 trains nine production models — seven dense sizes and two MoE variants — and tuning the learning rate $\mu$ and batch size $B$ by trial at 72B scale is not affordable. Instead the team ran a sweep of small and medium runs, dense models from 44M to 14B parameters and MoE models from 44M to 1B activated parameters, on 0.8B to 600B tokens, found the optimal $\mu_{\text{opt}}$ and $B_{\text{opt}}$ at each point, and fit how those optima move with $N$ and $D$. The fit is then extrapolated to the production sizes.

With optimal hyper-parameters predicted, the same data supports a second fit: final loss as a function of architecture and data scale. That is what makes the MoE design tractable. The team uses it to predict how MoE models with various total and activated parameter counts compare with dense models, and picks configurations that land at parity with specific dense models such as Qwen2.5-72B and Qwen2.5-14B — the positioning of Plus and Turbo.

One caveat is worth stating plainly: the report describes this methodology but publishes no fitted coefficients, exponents or curves. Readers can reproduce the approach, but not check the numbers.

Training on long sequences from the start is wasteful: attention cost grows with sequence length and most pre-training knowledge does not need 32K tokens of context. Qwen2.5 therefore pre-trains in two phases. The bulk of training runs at 4,096 tokens; the final stage extends every model except Turbo to 32,768 tokens.

Extending context is mostly a positional-encoding problem. RoPE rotates query and key pairs at frequencies $\theta_i = b^{-2i/d}$; with the default base $b = 10{,}000$ the slowest dimensions complete their rotation well before 32K tokens, so distant positions alias with nearby ones. Qwen2.5 applies ABF (adjusted base frequency) and raises the base from 10,000 to 1,000,000 during the extension stage. A larger base stretches every wavelength, so the model sees distinct rotations across the longer window.

Qwen2.5-Turbo targets 1M tokens and goes further with a progressive four-stage schedule: 32,768 → 65,536 → 131,072 → 262,144 tokens, with RoPE base 10,000,000. At each stage the data is 40% sequences at the current maximum length and 60% shorter sequences, so the model adapts to longer inputs without losing its behaviour on short ones.

Training stops at 32K (or 262K for Turbo), but inference goes further. YaRN rescales RoPE frequencies at inference time, and Dual Chunk Attention (DCA) splits a long sequence into chunks and remaps relative positions so no query–key distance exceeds what was seen in training. Together they give a four-fold increase: open models reach 131,072 tokens and Turbo reaches 1 million. Both are training-free and leave behaviour within 32K unchanged, which is why the RULER table's "w/o DCA + YARN" rows match the full rows up to 32K and diverge only beyond it.

Post-training in Qwen2.5 is a three-step pipeline — SFT, then offline RL, then online RL — and SFT carries most of the data. The team built over 1 million SFT examples, organised around the specific places Qwen2 fell short.

The most user-visible gap was length. Typical post-training responses stay under about 2,000 tokens, so models learn to stop early. Qwen2.5 targets 8,192-token outputs by building long-response data: it back-translates long pre-training documents into the queries that would have produced them, imposes output-length constraints, and filters weak pairs with Qwen2.

The remaining areas share one idea: verify before you train. Math uses Qwen2.5-Math chain-of-thought data with rejection sampling guided by a reward model and reference answers. Code uses Qwen2.5-Coder's multi-agent instruction data across nearly 40 programming languages, checked in a multilingual sandbox with static analysis and unit tests. Instruction-following data is generated together with verification code and unit tests, and kept only if execution feedback confirms compliance. Logical reasoning adds 70,000 new queries; structured-data understanding adds tables and semi-structured inputs with reasoning chains; cross-lingual data is translated and then checked for semantic alignment; and hundreds of system prompts make the model robust to how it is instructed. A final response filter keeps only outputs judged flawless by both a critic model and a multi-agent scoring system.

Training is short and conservative: two epochs at 32,768-token sequences, learning rate decayed from $7 \times 10^{-6}$ to $7 \times 10^{-7}$, weight decay 0.1, gradient clipping at 1.0.

Qwen2.5 splits reinforcement learning into two stages because reward models are good at some things and bad at others. A learned reward model can judge whether a response is helpful, concise or harmless. It is much less reliable at judging whether a proof is valid, a program is correct, or a response obeys a formatting constraint. Those domains — mathematics, coding, instruction following and logical reasoning — have checkable answers, so a reward model is unnecessary there.

Offline RL handles them. The SFT model resamples responses for a new set of queries, and the same execution-feedback and answer-matching pipeline from SFT grades them. Responses that pass become positive examples; responses that fail become negative examples. Human and automated review then filter the pairs. The result is roughly 150,000 preference pairs, trained with Direct Preference Optimization for one epoch using the Online Merging Optimizer at learning rate $7 \times 10^{-7}$.

The design choice is to build reliable training signals ahead of time instead of computing them during training. DPO needs no reward model and no rollouts, only pairs. Because the pairs are graded by execution and answer matching rather than a learned scorer, the policy cannot exploit scorer errors here — those errors are the main risk in the online stage.

The online stage covers what offline RL cannot: qualities that have no reference answer but that a good reward model can detect. The reward model is trained against six labeling criteria — truthfulness, helpfulness, conciseness, relevance, harmlessness and debiasing. Its training queries come from open-source data and a harder proprietary set; responses are sampled from Qwen checkpoints at different stages (SFT, DPO, RL) and at different temperatures, labelled by humans and automated processes, and the DPO pairs from the offline stage are mixed in.

The policy is optimised with Group Relative Policy Optimization. For each query GRPO samples a group of responses — 8 per query in Qwen2.5 — scores each with the reward model, and uses each response's score relative to its group as the advantage. No value network is needed, because the group mean is the baseline.

Qwen2.5 adds one scheduling idea. Queries are ordered by the variance of their response scores under the reward model, and higher-variance queries are processed first. The reasoning follows from the advantage formula: if all 8 responses score about the same, every normalised advantage is near zero and the query produces almost no gradient. Queries where the samples disagree are the ones the policy can learn from. The query set for RL is the same one used to train the reward model, and training uses a global batch size of 2,048 with 2,048 samples per episode.

Because the online stage depends on it, the report evaluates Qwen2.5-RM-72B separately. Table 15 compares it with Nemotron-4-340B-Reward, Llama-3.1-Nemotron-70B-Reward and Athene-RM-70B on four benchmarks: Reward Bench, RMB, PPE, and an internal out-of-domain Human-Preference-Chinese set.

No model wins everywhere. Llama-3.1-Nemotron-70B-Reward leads Reward Bench (94.10) and Athene-RM-70B leads RMB (73.98). Qwen2.5-RM-72B leads PPE's objective average (69.85) and Human-Preference-Chinese (61.27), comes second on RMB (68.71), and scores 91.59 on Reward Bench — close to Nemotron-4-340B's 92.00.

The authors draw two conclusions. First, Reward Bench has become the default yardstick, and over-optimising for it looks like Goodhart's law: models that excel there fall back on other benchmarks, which may hurt downstream alignment. Athene-RM-70B scores 88.32 on Reward Bench yet leads RMB. Second, and more important, reward-model benchmark scores did not predict how well the resulting RL policy performed. A higher RM score did not necessarily produce a better RL model. The authors present this as an open research problem, not a solved one.

Pre-training gives Qwen2.5-Turbo the positional range for 1M tokens. Two further problems remain: following instructions over very long inputs, and responding before the user gives up.

For alignment, Turbo's SFT runs in two stages. Stage one uses only short instructions (up to 32,768 tokens) with the same data and steps as the other Qwen2.5 models, which preserves short-task quality. Stage two mixes short instructions with long instructions of up to 262,144 tokens. RL, however, stays short-only. The report gives two reasons: RL on long contexts is computationally expensive, and few reward models can give reliable signals on long inputs. It also reports a useful result: RL on short instructions alone still noticeably improves alignment on long-context tasks.

Figure 2 is the retrieval check: Turbo reaches 100% accuracy on 1M-token passkey retrieval at every context length and document depth tested. Figure 3 addresses speed. Full attention over 1M tokens is very slow to prefill, so Qwen2.5 builds a sparse attention mechanism based on MInference that cuts attention computation by 12.5x at 1M tokens. Time to first token falls by 3.2x to 4.3x for Turbo at 1M, depending on hardware (4.3x on H20, 3.2x on A100), and by up to 5.6x for Qwen2.5-7B.

The passkey result should be read in context. Passkey retrieval is the easiest long-context test, since it only requires finding one inserted number. RULER and LV-Eval, covered under Results, are harder tests of whether the model can reason over the whole context.

The headline claim is a parameter-efficiency claim: Qwen2.5-72B-Instruct is competitive with Llama-3.1-405B-Instruct, a model about 5.6x larger. The abstract says "around 5 times larger" and the conclusion says "six times smaller"; the actual ratio $405 / 72 \approx 5.6$ falls between them.

Base models (Table 2). Qwen2.5-72B scores 86.1 MMLU, 62.1 MATH, 91.5 GSM8K and 84.7 MBPP, against Llama-3-405B's 85.2, 53.8, 89.0 and 73.0. It is not better on every row: on HumanEval it scores 59.1, below Qwen2-72B (64.6) and Llama-3-405B (61.0). Qwen2.5-Plus reaches 64.0 MMLU-Pro, 5.9 points above Qwen2.5-72B.

Instruction-tuned models (Table 6). Qwen2.5-72B-Instruct beats Llama-3.1-405B-Instruct on MMLU-redux (86.8 vs 86.2), MATH (83.1 vs 73.8), MBPP (88.2 vs 84.5), MultiPL-E (75.1 vs 73.5), LiveCodeBench (55.5 vs 41.6), Arena-Hard (81.2 vs 69.3) and MT-Bench (9.35 vs 9.08). It trails on the other six: MMLU-Pro (71.1 vs 73.3), LiveBench (52.3 vs 53.2), GPQA (49.0 vs 51.1), GSM8K (95.8 vs 96.8), HumanEval (86.6 vs 89.0) and IFEval (84.1 vs 86.0). Qwen2.5-Plus beats Qwen2.5-72B-Instruct on 9 of 13 benchmarks, which matches the table.

Smaller models. Qwen2.5-7B-Instruct scores 75.5 on MATH and 84.8 on HumanEval, against Gemma2-9B's 44.3 and 68.9. Qwen2.5-3B-Instruct reaches 65.9 MATH with 2.8B non-embedding parameters. One claim does not match its table: the report says Qwen2.5-Turbo beats Qwen2.5-14B-Instruct on "eight out of ten benchmarks," but Table 7 has 13 rows and Turbo leads on 6 of them (MMLU-Pro, MMLU-redux, MATH, HumanEval, MBPP, MultiPL-E).

Long context (Table 16). Qwen2.5-72B-Instruct averages 95.1 on RULER and scores 88.4 at 128K, ahead of GPT-4 (81.2) and Llama-3.1-70B (66.6). The ablation rows isolate YaRN + DCA: without them, 72B falls to 67.0 at 128K and 7B to 31.4 (from 55.1). On LV-Eval at 256K, 72B scores 45.2 with the extension methods and 2.4 without.

All evaluations use the same decontamination rule as Qwen2: a training sequence is removed if its longest common subsequence with any test sequence is at least 13 tokens and covers at least 60% of the shorter sequence.