Search-R1

A language model's knowledge stops at its training cutoff and is patchy on anything long-tail, so a question like "Curious is a women's fragrance by a singer born in what city and state?" is a trap. It needs two facts chained together, and the model is unlikely to hold both reliably. The obvious fix is to give it a search engine. How it should use that engine is the hard part.

Before Search-R1 there were two families of answer. Retrieval-augmented generation (RAG) runs one retrieval using the user's question as the query and pastes the passages into context. That is cheap and robust, but the query is fixed before the model has thought about anything, so it suits single-hop questions and can fall apart on multi-hop ones, where the second query depends on what the first one returned. Search-as-a-tool methods let the model issue its own queries mid-reasoning. IRCoT and ReAct do this with prompting, which the paper calls suboptimal because the LLM was never optimised to interact with a search engine. Toolformer does it with supervised fine-tuning, which needs large sets of high-quality annotated search trajectories that are expensive to produce at scale. And search is non-differentiable, so you can't simply backpropagate through it.

Reinforcement learning sidesteps both problems. DeepSeek-R1 showed that RL from outcome rewards alone can teach a model to reason, with no annotated chains of thought. Search-R1 asks whether the same recipe can teach a model when and what to search. The authors identify three challenges: (1) framework and stability — how to put a search engine inside an RL loop when part of the trajectory is text the model did not write; (2) multi-turn interleaving — the model should decide dynamically how many searches a question needs; and (3) reward design — whether a simple exact-match outcome reward is enough to induce meaningful search behaviour.

The paper answers all three, and describes itself as an extension of DeepSeek-R1-Zero from parametric reasoning to retrieval-driven reasoning.

Standard RL fine-tuning of an LLM (RLHF and its descendants) maximises a reward on responses sampled from the policy, minus a KL penalty that keeps the policy close to a frozen reference model. It assumes the entire response $y$ is written by the policy. Search-R1 breaks that assumption on purpose: during rollout, the model's text is interleaved with passages returned by a search engine $\mathcal{R}$.

The paper formalises this by conditioning the policy on the engine: trajectories are drawn from $\pi_\theta(\cdot \mid x; \mathcal{R})$, which it describes as $\pi_\theta(\cdot \mid x) \bigotimes \mathcal{R}$, interleaved retrieval-and-reasoning. The objective keeps the familiar shape (reward minus KL), but both the expectation and the KL term are now taken over these search-augmented trajectories. The search engine is treated as part of the environment. It is frozen, it is never trained, and it only produces observations.

Figure 1 shows the framework is compatible with both mainstream policy-gradient methods. PPO uses a learned value model and Generalized Advantage Estimation to give a per-token advantage. GRPO drops the value model and computes advantages by comparing a group of rollouts for the same question. In both cases the rollout module is the same: a policy LLM that can call the search engine as many times as it decides to, up to a budget. The reward model and reference LLM are frozen.

Treating retrieval as environment feedback rather than as model output is what makes the design work. It is also what forces the next idea, loss masking: if retrieved tokens are observations, the policy-gradient loss should not be computed on them.

During rollout the policy is not simply sampling one long completion. Algorithm 1 runs a small generate–detect–retrieve loop. The LLM generates tokens until it emits one of three stop signals: </search>, </answer>, or end-of-sequence. What happens next depends on which one it was.

If the segment contains a <search> query </search> block, the system parses out the query, calls the search engine, and appends the top results to the trajectory wrapped in <information> … </information>. The LLM then keeps generating, now conditioned on what it just retrieved. If the segment contains <answer> … </answer>, the rollout ends and that answer is scored. If the model produced neither, which is a malformed action, the system appends the fixed string "My action is not correct. Let me rethink." and lets it try again.

Each pass through the loop costs one unit of an action budget $B$ (set to 4 in the experiments). Search depth therefore isn't fixed by the pipeline. The model decides it, from zero searches on an easy question up to the budget on a hard multi-hop one, and RL rewards whichever choice yields a correct answer. The retriever is E5 over a 2018 Wikipedia dump, returning the top 3 passages per query, with retrieved content capped at 500 tokens.

This is the mechanism behind the paper's case studies: a model that queries "Curious fragrance information", learns it is a Britney Spears perfume, then queries "Britney Spears birthplace". No single-shot RAG query could have chained those two steps.

RL can only reinforce behaviour the model sometimes produces, so the starting policy needs some way of expressing a search. Search-R1 supplies it with a single instruction template (Table 1), and very little else.

The template tells the model to reason inside <think> tags every time it gets new information, to call the search engine with <search> query </search> if it lacks knowledge, that results will come back between <information> tags, that it can search as many times as it wants, and to put the final answer inside <answer> tags without detailed illustrations. Then it appends the question.

What it leaves out is the point. The authors limit constraints to structure: the template does not demand reflective reasoning, does not require a search on every question, and does not endorse any problem-solving strategy. That keeps the model's learning dynamics during RL observable and unbiased. Behaviours that emerge later, such as decomposing a multi-hop question, re-querying after a bad result, or running a verification search, are learned from reward and were not prompted.

The same template is used for base and instruct models. A base model barely follows it at first, which is one reason base models start from a lower reward. As later sections show, RL closes that gap.

A Search-R1 trajectory is a mix of two kinds of tokens: those the policy generated (thoughts, queries, the answer) and those the search engine returned (the passages inside <information>). Standard PPO and GRPO compute a token-level loss over every token in the rollout. Applied naively here, that would push gradients through the passages too.

That is wrong on two counts. The policy never chose those tokens, so increasing or decreasing their likelihood based on the trajectory's reward assigns credit to text the agent did not produce. It also teaches the model to predict Wikipedia rather than to use it, an objective unrelated to searching well, which the paper describes as unintended learning dynamics.

The fix is a one-line indicator. $I(y_t) = 1$ if token $y_t$ was LLM-generated and $0$ if it was retrieved. Both the PPO and GRPO objectives sum only over tokens with $I(y_t) = 1$ and normalise by their count, and in GRPO the KL penalty is masked the same way. The retrieved passages still sit in the context and condition every subsequent token, but they never receive gradient.

The ablation is decisive. On Qwen2.5-7B-base with PPO, masking lifts the seven-dataset average EM from 0.343 to 0.431, and it wins on every dataset. Figure 3 shows higher training reward on both 3B and 7B, and the 3B average rises from 0.262 to 0.303.

The training signal is about as simple as it can be. When a rollout finishes, the system extracts the text inside <answer> and checks it against the gold answer with exact match. A match scores 1 and anything else scores 0. Nothing rewards a well-formed query, a relevant retrieval, a sensible number of searches, or the format.

Two omissions are deliberate. First, unlike DeepSeek-R1, there is no format reward. The authors report that the learned model already adheres strongly to the tag structure, and leave richer format rewards to future work. Second, there is no neural reward model. Following DeepSeek-R1, they avoid one because LLMs in large-scale RL are sensitive to the specific form of the reward (a learned reward model can be gamed), and because training one adds compute and complexity.

This is challenge (3) from the introduction: is an outcome reward enough to induce good search behaviour? The rest of the paper says yes. With nothing but EM on the final answer, the model learns to issue more valid searches over training, to decompose multi-hop questions into sequential queries, and sometimes to run a self-verification search before answering. All of that credit assignment flows backward from a single bit at the end of the trajectory.

Search-R1 runs under either algorithm, so the paper compares them directly on Qwen2.5-3B and 7B, base and instruct. GRPO differs from PPO by using the average reward of a group of sampled responses as its baseline, rather than a learned value function: for each question it samples $G$ rollouts (5 by default) and scores each one relative to the others.

The paper reports three findings. GRPO converges faster in every setting, because PPO's critic needs warm-up steps before its advantage estimates are useful. PPO is more stable: in Figure 5, GRPO's training reward collapses after many steps on several models, while PPO's does not. And final rewards are comparable, so both are viable. The authors choose PPO as the default because of its stability, and evaluate collapsed runs at their last stable checkpoint.

The evaluation numbers (Table 3) show no single winner. On 7B-base, PPO scores 0.431 average EM against GRPO's 0.350. On 7B-instruct, GRPO leads, 0.396 to 0.385. On 3B, GRPO is ahead for both base (0.312 vs 0.303) and instruct (0.336 vs 0.325).

The group-size ablation (Table 8) sharpens the trade-off. Larger groups converge faster and reach higher training reward but risk collapse. Group size 1, where GRPO reduces to plain REINFORCE, was the most stable and generalised best, at 0.410 average EM against 0.350 for size 5.

The evaluation covers seven QA benchmarks in two groups: general QA (NQ, TriviaQA, PopQA) and multi-hop QA (HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle). Only NQ and HotpotQA are in-domain, since their training sets were merged for training; the other five are out-of-domain. Every baseline uses the same retriever, the same top-3 passages, the same corpus, the same training data and the same base LLMs, so the comparison isolates the method.

The baselines cover the alternatives: no retrieval (direct inference, CoT); retrieval at inference time (RAG, IRCoT, Search-o1); and fine-tuning (SFT, R1 — the same RL recipe without a search engine — and rejection sampling, which fine-tunes on the policy's own correct search trajectories).

On Qwen2.5-7B, Search-R1-base reaches 0.431 average EM against 0.348 for rejection sampling, the strongest baseline, and 0.304 for RAG, with the largest gains on multi-hop sets (HotpotQA 0.433 vs RAG's 0.299). On Qwen2.5-3B, Search-R1-instruct reaches 0.325 against RAG's 0.270. The paper's headline numbers are not consistent: the abstract claims 24% (7B) and 20% (3B) average relative improvement over RAG baselines, while the contributions list claims 41% and 20%. Computed directly from Table 2 against RAG, the gains are about 42% and 20%. Search-R1 also beats R1 at every size, which isolates the value of search from the value of RL. Appendix C repeats the study on Qwen2.5-14B, where Search-R1-base reaches 0.479, and the gap widens with scale.

The paper's fourth observation is that larger models learn search better: Search-R1's margin over the second-best method is much bigger at 7B than at 3B. One anomaly appears in the 3B-base row, where Bamboogle EM is 0.088, far below the 3B-instruct model's 0.264.

Figure 2 tracks how Search-R1 changes over training, and it describes a learning process rather than a gradual climb.

Response length (Qwen2.5-7B-base, panel c) follows a decrease–increase–stabilise pattern. For roughly the first 100 steps it falls sharply while reward rises only slightly: the base model is dropping filler words and adapting to the task format. After about step 100, both length and reward rise significantly. That is the point where the model starts calling the search engine frequently, and the retrieved passages lengthen the trajectory. Panel d shows the same story from the other side: the number of valid search calls grows steadily over training. Nobody told the model to search more. It learned that searching pays.

Base vs instruct (panel b, Figure 4): instruction-tuned models start higher and converge faster, but final rewards are very similar. General post-training speeds up early learning in this setting, and RL can close the gap for base models over time.

How many passages to retrieve (Appendix G, Table 7) shows a precision–recall trade-off inside RL. Top-5 converges fastest but its reward later declines and becomes unstable; top-1 is stable but limited by recall. Top-3 is best, at 0.431 average EM against 0.375 for top-1 and 0.400 for top-5. The authors suggest that noisy passages at top-5 may teach the model that retrieved context is often unhelpful, discouraging it from relying on search.

The appendix compares a model trained with RL without search (R1) and Search-R1 on the same question (Table 9): "Curious is a women's fragrance by a singer born in what city and state?" R1 reasons from memory: "The singer is Beyoncé, who was born in Houston, Texas," and answers Houston. It is fluent, confident and wrong.

Search-R1 searches "Curious fragrance information" and learns the perfume is by Britney Spears. It then searches "Britney Spears birthplace" and finds McComb, Mississippi. The paper points to two behaviours here. One is interleaved reasoning and retrieval: each query is written using what the previous one returned. The other is self-verification: after the second search the model already has the answer, yet it issues a third search, "McComb, Mississippi location", to confirm it before answering. This mirrors the self-verification that DeepSeek-R1 observed in pure-reasoning RL.

The paper's eleven additional case studies (Appendix J) are candid about failures, and they map onto the method's limits:

• Answer-span errors under EM. For the Weezer question, the model retrieves the right passage and even names "The Blue Album" in its reasoning, then answers "Weezer".
• Poor decomposition. It sometimes searches the whole question verbatim, repeats an identical query that already failed, and then guesses ("National Magazine Award" instead of Pulitzer Prize).
• Misled by irrelevant retrievals. A passage about a different reality show leads to "London Charles" instead of Mani.

Each of these is a case where one EM bit at the end gives weak guidance, which is why the conclusion names richer reward mechanisms and uncertainty-aware retrieval as next steps.