v9.1.0
Menu
Sign In

Chapter 3: Self-Improvement: Reflexion, Self-Refine, Tree-of-Thought, LATS

Week 3-4

Agents that learn from their own mistakes — without weight updates. Reflexion's verbal reinforcement learning, Self-Refine's critic loops, Tree of Thoughts' deliberate search over reasoning paths, and Language Agent Tree Search (LATS) which combines them all.

Topics
5
Demos
2
Min read
15

Chapter Overview

A bare ReAct agent that fails has no way to recover. The next time it sees a similar problem, it makes the same mistake. Self-improvement techniques fix this without retraining the model — they let the agent inspect its own trajectory, write a critique, and use the critique as additional context on the next attempt.

Four techniques dominate:

  1. Reflexion (Shinn et al., 2023) — verbal reinforcement learning. After a failed attempt, the model writes a paragraph of self-critique into an episodic memory. The next attempt sees the critique in its prompt.
  2. Self-Refine (Madaan et al., 2023) — single-shot critique-and-revise. The model produces an answer, then critiques its own answer, then revises based on the critique. Iterate until convergence.
  3. Tree of Thoughts (Yao et al., 2023) — explore the tree of reasoning paths instead of committing to a single chain. Use BFS/DFS with an LLM-as-judge to score branches.
  4. Language Agent Tree Search (LATS) (Zhou et al., 2023) — unify ReAct, Reflexion, and ToT under MCTS. The state-of-the-art for hard reasoning + acting tasks.

The meta-insight: for many tasks, more inference-time compute on a smaller model beats more training-time compute on a larger model. Self-improvement is the most effective way to spend that inference compute.

This chapter covers:

  • Reflexion: Verbal RL — the loop of attempt → reflect → retry
  • Self-Refine & Critic loops — generate, critique, revise
  • Tree of Thoughts — search over reasoning paths with LLM-as-judge
  • LATS — MCTS over agent trajectories
  • Actor / Critic / Reflection tradeoffs — what each technique buys you, what it costs

Chapter Roadmap

Click any topic to jump in

Interactive demo
1
Reflexion (Verbal RL)

Write a paragraph of self-critique, feed it into the next attempt. Gradient-free policy improvement.

The Three-Component LoopWhy Verbal RL Beats Numerical RL Here
Two ways to add a critic in the loop

Same trajectory (Self-Refine) or search the tree (ToT)

2
Self-Refine

Generate, critique, revise — single-shot self-improvement for one-off generation tasks.

The Generate / Critique / Revise CycleSelf-Refine vs Reflexion vs LLM-as-Judge
3
Tree of Thoughts

Search the tree of reasoning paths with LLM-as-judge scoring.

Branching, Scoring, PruningWhen ToT Pays for Itself
MCTS unifies all the prior pieces
4
LATS (MCTS Unifier)

MCTS over agent trajectories — combines ReAct, Reflexion, and ToT into one framework.

MCTS over Agent TrajectoriesWhy LATS Wins on Hard Tasks
The cost-quality frontier crystallizes into a priority order
5
Where to Spend Compute

A priority order for inference-time techniques when your agent is failing.

The Inference-Time Compute FrontierWhere to Spend Compute First

Topics

5 topics in this chapter

A bare ReAct agent that fails has no memory of the failure: face it with a similar problem tomorrow and it repeats the same mistake, because nothing in its loop carries a lesson from one attempt to the next. Chapter 2 gave you better plans, but a good plan still fails against a surprising world. Reflexion (Shinn et al., 2023) is the first and simplest self-improvement technique, and it fixes recovery without touching a single weight. After a failed attempt the agent writes a paragraph of natural-language self-critique — an episodic memory of what went wrong — and that paragraph is prepended to the next attempt's prompt. This topic builds the three-role loop of Actor, Evaluator, and Self-Reflector that makes it work, and explains why it is called verbal reinforcement learning: it is policy improvement carried out in prompt space rather than parameter space. The reason it works at all is that strong LLMs learn from specific textual feedback far more readily than they plan a flawless trajectory on the first try — a theme that runs through every technique in the chapter.

Definition

Reflexion is a self-improvement loop in which an Actor produces a trajectory, an Evaluator scores it, and a Self-Reflector writes a natural-language critique of any failure into episodic memory, which is fed into the next attempt's prompt. It improves the policy through in-context textual feedback rather than weight updates — verbal reinforcement learning.

In this topic

  1. 1The Three-Component Loop
  2. 2Why Verbal RL Beats Numerical RL Here
1 of 2
The Three-Component Loop

promptn+1=system ∥ goal ∥ [rn−2,rn−1,rn]⏟reflections from last 3 attempts\text{prompt}_{n+1} = \text{system} \,\Vert\, \text{goal} \,\Vert\, \underbrace{[r_{n-2}, r_{n-1}, r_n]}_\text{reflections from last 3 attempts}

Reflexion factors the agent into three roles, each played by an LLM call:

  1. Actor. A standard ReAct agent that takes actions in the environment. Outputs a trajectory.
  2. Evaluator. Examines the trajectory and emits a binary or scalar score (did the task succeed? how well?). Can be a programmatic check (test passed?), an LLM-as-judge call, or a human.
  3. Self-Reflector. Given a failed trajectory and the evaluator's score, writes a paragraph of natural-language reflection: "I failed because I forgot to escape the SQL string. Next time, sanitize all user input before query construction."

The reflection is appended to an episodic memory buffer. On the next attempt, the actor's prompt includes the most recent reflections (typically the last k=3k=3). The actor reads the reflections and adjusts its strategy.

Mathematical Intuition

Reflexion's improvement can be framed as policy improvement under a textual representation. The reflection rnr_n is a hint sampled from P(r∣τn,Rn)P(r \mid \tau_n, R_n) — the conditional distribution over hints given the failed trajectory. The next attempt's policy is πn+1(a∣s)=πθ(a∣s,rn−2:n)\pi_{n+1}(a \mid s) = \pi_\theta(a \mid s, r_{n-2:n}). This is policy iteration in prompt space rather than parameter space — gradient-free, but it works because the in-context conditioning of strong LLMs is approximately as expressive as a small fine-tune.

Example:

An agent attempts a coding task and fails the test suite. What does Reflexion add to the next attempt?

2 of 2
Why Verbal RL Beats Numerical RL Here

Classical RL gives the policy a scalar reward and updates parameters via backprop. For agentic tasks this is hard:

  • The reward signal is sparse (only at the end of the trajectory)
  • Credit assignment across 50+ steps is brittle
  • Off-policy data is expensive to collect

Verbal RL sidesteps all of this. The reflection IS the credit assignment — written in natural language by an LLM that is much better at counterfactual reasoning ('what if I had done X instead?') than a value function is at backing up scalars. And because the reflection is text, it can be highly specific to the task: 'don't forget to escape SQL strings' is a much more useful signal than 'reward = -0.3'.

Papers
Reflexion: Language Agents with Verbal Reinforcement LearningSelf-Refine: Iterative Refinement with Self-Feedback
Blogs
Reflexion — Lilian Weng's Agent Survey

Related Problems on PixelBank

Reflexion Retry DecisionPRO