WMRL — World Model RL
Scaling Automatic Research Agents by Replacing Environment Execution with a World Model
Xiyuan Yang, Sheikh Sarwar, Jingru Cheng, Zhan Shi, Duanshun Li, Huiyuan Chen, Haiyang Zhang, Xing Fan, Chenlei Guo, Jingrui He, Zhenyu Liao
Read the Paper on arXivPaper Overview
WMRL (Yang, Sarwar, Cheng et al. — University of Illinois Urbana-Champaign & Amazon, 2026) starts from a cost-accounting observation about training research agents and turns it into a scaling result. An AutoResearch agent is a language model that conducts empirical research on its own: given a question, it writes an experiment, runs it, reads the outcome, and iterates. Reinforcement learning is the natural way to make such agents better, because the execution outcome is a reward — but RL needs scale, and scaling reveals a structural asymmetry.
Every AutoResearch trajectory has two halves. Generation (the agent writing code) is served by batched inference backends like vLLM and SGLang, where concurrent trajectories share compute, so an extra trajectory is nearly free. Execution (actually running the code) cannot be batched — each candidate solution needs its own isolated sandbox loading data and training models on real GPUs. So execution cost grows linearly with the number of trajectories and, past some point, dominates everything. Execution is the bottleneck.
WMRL's move is to replace environment execution with a world model — a language model prompted to predict the execution outcome in a few forward passes. Prediction batches and amortizes exactly like generation, so the bottleneck disappears. But the prediction can be wrong, and the paper is unusually honest about the bill: an imperfect world model corrupts the reward with a bias and a noise, each of which puts an extra term into the RL convergence bound.
The fix is a thin anchor signal — about 10% of groups are still graded by real execution. Online Debiasing fits a monotone map to erase the bias; Inverse-Variance Denoising fuses the cheap-noisy and scarce-clean gradient streams to suppress the noise. Both are proven to strictly improve convergence, and empirically WMRL trains 3-4x faster than full real-execution RL while matching or beating it — with post-trained 4B and 9B agents outrunning off-the-shelf 48B and 120B agents.
Chapter Roadmap
Click any topic to jump in
The Execution Bottleneck
Generation batches and amortizes; execution needs one exclusive sandbox per solution — so as trajectories grow, execution dominates the RL training cost.
World Model as Environment
Swap the real sandbox for a language model that predicts the execution outcome in a few forward passes — now execution scales as gracefully as generation.
The Price: Bias & Noise
An imperfect world model corrupts the reward with a systematic bias b and a zero-mean noise xi, which enter the convergence bound as two extra error terms.
Anchor Signal & Two Corrections
Keep ~10% of groups graded by both the world model and real execution. Online Debiasing removes b by isotonic regression; Inverse-Variance Denoising fuses the two streams to kill xi.
Convergence Guarantee
Theorem 4 divides the bias term by (1 + T/T0) and the variance term by (1 + V_WM/V_E) — the permanent error floor of pure world-model RL becomes a contracting term.
Evidence and transfer
Results
3.1x-3.4x less training compute than real-execution GRPO while scoring higher, and post-trained 4B / 9B agents beat off-the-shelf 48B / 120B agents.
Beyond AutoResearch
The same recipe transfers to embodied VLA post-training on LIBERO-Long, lifting overall success rate by 3.8 points — wherever execution, not generation, is the bottleneck.
Training a strong AutoResearch agent with RL means collecting a massive number of online trajectories, and each trajectory is two things stitched together: the agent generating a candidate solution, and the environment executing it to produce a reward. The paper's core observation is that these two halves scale in completely different ways.
Generation is served by modern inference backends where batching lets concurrent rollouts share compute. Add another trajectory and the marginal cost is nearly nothing — the GPUs were already busy, the new sequence just rides along. Execution has no such luck. Every candidate solution must run in its own isolated sandbox that loads the dataset and trains a model on real, exclusively-assigned GPUs. There is no batching to hide behind: each additional trajectory pays the full execution cost, and the total grows linearly with the number of trajectories.
The consequence is an asymmetry that only gets worse as you scale. Generation compute stays cheap per-trajectory; execution compute climbs until it hits the cluster's capacity and becomes the hard limit on how much RL you can run. Figure 1(c) makes it visual — in traditional RL the execution curve is what saturates first, capping the number of trajectories you can afford. This is the bottleneck WMRL exists to remove, and it motivates the two questions that structure the whole paper: (i) can we replace the expensive environment with a fast, scalable signal? and (ii) there is no free lunch — what does that signal cost, and how do we pay?
Key Points
Every AutoResearch trajectory splits into agent generation (writing a solution) and environment execution (running it to get a reward)
Generation batches: inference backends (vLLM, SGLang) let concurrent rollouts share compute, so an extra trajectory is nearly free
Execution does not batch: each candidate needs an isolated sandbox loading data and training on real GPUs — cost grows linearly with trajectory count
As RL scales up trajectory volume, execution compute saturates first and becomes the dominant bottleneck (Figure 1c)
This frames the paper's two questions: replace the expensive environment with a scalable signal, and then account for what that replacement costs
The reward in these pipelines comes from executing the agent's solution — so the bottleneck is inherent to execution-driven RL, not to any particular RL objective