Dream-RSI
Recursive Self-Improvement through Evolving Worlds
Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo
Read the Paper on arXivPaper Overview
Dream-RSI (Zheng, Wu, Zhang et al. — Google, Google DeepMind, University of Maryland & University of Virginia, September 2026) attacks a bottleneck that appears once AI agents start improving themselves. In recursive self-improvement (RSI), an agent runs a discovery loop — propose a solution, evaluate it, incorporate feedback, refine — and modern discovery problems push that loop across thousands of proposal–evaluation cycles. At that scale the deciding factor is no longer the coding agent but the exploration strategy: which branches to pursue, which candidates to refine, how much to run in parallel, when to stop.
Improving that strategy is caught in a dilemma. Hand-designed strategies stay fixed and cannot learn from accumulated experience. Optimizing them online is brutally expensive: unlike scoring one candidate, scoring an exploration policy requires watching it shape a whole discovery run over many cycles — delayed, costly feedback — and the space of policies is vast, so many must be tried.
The paper's key insight is that the feedback you need has already been paid for. A completed discovery run records a structured tree of past decisions and their realized code-execution outcomes. Organized as a tree, that history becomes a replay simulator: an alternative policy can traverse different branches, in different orders, with different parallel groupings, and be scored by simply revealing outcomes already stored — no agent, no evaluator rerun. By analogy to model-based RL and World Models, Dream-RSI lets the agent “dream” over these worlds to refine its exploration policy off-policy, then redeploys the improved policy online to gather new experience and grow the simulator pool — closing an RSI loop at the meta-exploration layer.
Across algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI matches or beats strong baselines while cutting discovery cost dramatically — up to 162x fewer discovery-agent calls than SimpleTES on the Lasso task, and 2.43x fewer generations on a VGG16 kernel at comparable performance.
Chapter Roadmap
Click any topic to jump in
The Meta-Exploration Bottleneck
Fixed strategies can't adapt as search spaces scale; optimizing them online means paying for a long rollout just to score one policy.
History as a Replay Simulator
A completed discovery tree already stores every node's outcome — so it can be replayed as a grounded, zero-execution-cost simulator.
Discovery Trees & the Decision Interface
Online generates new stochastic children; replay reveals recorded children deterministically. Both share one decision interface.
The Dreaming RSI Loop
Explore online, build a simulator, dream up and score M candidate policies cheaply, redeploy the best — and grow the pool.
The Replay Objective
Score a replayed policy by best quality, minus execution cost, plus a parallelism bonus — with a no-worse-than guarantee.
Evidence and behavior
Results
Across 8 tasks in three domains, competitive or better discovery quality at up to 162x fewer agent calls.
What the Policy Learns
Interactive replay beats prompt-level guidance, and the learned policy conserves then re-expands compute as progress plateaus.
Recursive self-improvement runs on a discovery loop: generate candidate solutions, evaluate outcomes, fold in feedback, refine. This loop has driven real progress — algorithm design, open-ended mathematical optimization, systems design, agent self-improvement — and its discoveries increasingly feed back into building more capable systems. But as targets get harder, discovery demands long-horizon exploration over vast search spaces, often thousands of proposal–evaluation cycles. At that scale, how you orchestrate exploration decides whether compute is spent well or wasted.
Most systems orchestrate with a fixed, hand-designed strategy that never changes during discovery. A fixed strategy cannot learn from what it has already seen and will happily re-allocate compute to search directions that already failed. The natural fix is to optimize the exploration policy online, mid-discovery. That runs straight into two structural problems.
First, feedback at the meta level is delayed and expensive. Scoring a single candidate is one evaluation. Scoring an exploration policy means watching how it shapes an entire discovery process over many cycles before you know whether it was any good. Second, the meta-policy space is vast: a freshly proposed policy may be poor, so many alternatives must be tried — each demanding its own long online rollout before it returns useful signal. Together these make closing the self-improvement loop at the exploration layer painfully slow.
The rest of the paper is one idea worked out carefully: if a cheap simulator of discovery existed, you could evaluate many policies before committing to a costly online deployment — and, surprisingly, completed discovery histories already are that simulator.
Key Points
RSI is an iterative discovery loop — propose, evaluate, incorporate feedback, refine — and hard targets push it across thousands of proposal–evaluation cycles
At that scale, orchestrating exploration (which branches, which refinements, how much parallelism, when to stop) is the deciding factor, not the coding agent
Fixed strategies can't learn from accumulated experience and re-spend compute on directions that already failed
Optimizing the policy online hits two walls: meta-feedback is delayed and expensive (a policy is judged only after a long rollout), and the meta-policy space is vast (many candidates must be tried)
A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged