Week 3-4

Chapter 3: Self-Improvement: Reflexion, Self-Refine, Tree-of-Thought, LATS

Agents that learn from their own mistakes — without weight updates. Reflexion's verbal reinforcement learning, Self-Refine's critic loops, Tree of Thoughts' deliberate search over reasoning paths, and Language Agent Tree Search (LATS) which combines them all.

Chapter Overview

A bare ReAct agent that fails has no way to recover. The next time it sees a similar problem, it makes the same mistake. Self-improvement techniques fix this without retraining the model — they let the agent inspect its own trajectory, write a critique, and use the critique as additional context on the next attempt.

Four techniques dominate:

Reflexion (Shinn et al., 2023) — verbal reinforcement learning. After a failed attempt, the model writes a paragraph of self-critique into an episodic memory. The next attempt sees the critique in its prompt.
Self-Refine (Madaan et al., 2023) — single-shot critique-and-revise. The model produces an answer, then critiques its own answer, then revises based on the critique. Iterate until convergence.
Tree of Thoughts (Yao et al., 2023) — explore the tree of reasoning paths instead of committing to a single chain. Use BFS/DFS with an LLM-as-judge to score branches.
Language Agent Tree Search (LATS) (Zhou et al., 2023) — unify ReAct, Reflexion, and ToT under MCTS. The state-of-the-art for hard reasoning + acting tasks.

The meta-insight: for many tasks, more inference-time compute on a smaller model beats more training-time compute on a larger model. Self-improvement is the most effective way to spend that inference compute.

This chapter covers:

Reflexion: Verbal RL — the loop of attempt → reflect → retry
Self-Refine & Critic loops — generate, critique, revise
Tree of Thoughts — search over reasoning paths with LLM-as-judge
LATS — MCTS over agent trajectories
Actor / Critic / Reflection tradeoffs — what each technique buys you, what it costs

Chapter Roadmap

Click any topic to jump in

Reflexion (Verbal RL)

Write a paragraph of self-critique, feed it into the next attempt. Gradient-free policy improvement.

The Three-Component LoopWhy Verbal RL Beats Numerical RL Here

Two ways to add a critic in the loop

Same trajectory (Self-Refine) or search the tree (ToT)

Self-Refine

Generate, critique, revise — single-shot self-improvement for one-off generation tasks.

The Generate / Critique / Revise CycleSelf-Refine vs Reflexion vs LLM-as-Judge

Tree of Thoughts

Search the tree of reasoning paths with LLM-as-judge scoring.

Branching, Scoring, PruningWhen ToT Pays for Itself

MCTS unifies all the prior pieces

LATS (MCTS Unifier)

MCTS over agent trajectories — combines ReAct, Reflexion, and ToT into one framework.

MCTS over Agent TrajectoriesWhy LATS Wins on Hard Tasks

The cost-quality frontier crystallizes into a priority order

Where to Spend Compute

A priority order for inference-time techniques when your agent is failing.

The Inference-Time Compute FrontierWhere to Spend Compute First

Reflexion (Shinn et al., 2023) showed that an agent can dramatically improve its success rate on multi-step tasks just by writing a paragraph of self-critique after each failed attempt and feeding that paragraph into the next attempt's prompt. No weight updates, no fine-tuning — just an episodic memory of what went wrong.

The technique is so simple it sounds like a magic trick. It works because LLMs are very good at learning from textual feedback in context — much better than they are at planning a perfect trajectory on the first try.

In this topic

1The Three-Component Loop

2Why Verbal RL Beats Numerical RL Here

1 of 2

The Three-Component Loop

$\text{prompt}_{n+1} = \text{system} \,\Vert\, \text{goal} \,\Vert\, \underbrace{[r_{n-2}, r_{n-1}, r_n]}_\text{reflections from last 3 attempts}$

Reflexion factors the agent into three roles, each played by an LLM call:

Actor. A standard ReAct agent that takes actions in the environment. Outputs a trajectory.
Evaluator. Examines the trajectory and emits a binary or scalar score (did the task succeed? how well?). Can be a programmatic check (test passed?), an LLM-as-judge call, or a human.
Self-Reflector. Given a failed trajectory and the evaluator's score, writes a paragraph of natural-language reflection: "I failed because I forgot to escape the SQL string. Next time, sanitize all user input before query construction."

The reflection is appended to an episodic memory buffer. On the next attempt, the actor's prompt includes the most recent reflections (typically the last $k=3$ ). The actor reads the reflections and adjusts its strategy.

Mathematical Intuition

Reflexion's improvement can be framed as policy improvement under a textual representation. The reflection $r_n$ is a hint sampled from $P(r \mid \tau_n, R_n)$ — the conditional distribution over hints given the failed trajectory. The next attempt's policy is $\pi_{n+1}(a \mid s) = \pi_\theta(a \mid s, r_{n-2:n})$ . This is policy iteration in prompt space rather than parameter space — gradient-free, but it works because the in-context conditioning of strong LLMs is approximately as expressive as a small fine-tune.

Example:

An agent attempts a coding task and fails the test suite. What does Reflexion add to the next attempt?

2 of 2

Why Verbal RL Beats Numerical RL Here

Classical RL gives the policy a scalar reward and updates parameters via backprop. For agentic tasks this is hard:

The reward signal is sparse (only at the end of the trajectory)
Credit assignment across 50+ steps is brittle
Off-policy data is expensive to collect

Verbal RL sidesteps all of this. The reflection IS the credit assignment — written in natural language by an LLM that is much better at counterfactual reasoning ('what if I had done X instead?') than a value function is at backing up scalars. And because the reflection is text, it can be highly specific to the task: 'don't forget to escape SQL strings' is a much more useful signal than 'reward = -0.3'.

Papers

Reflexion: Language Agents with Verbal Reinforcement Learning Self-Refine: Iterative Refinement with Self-Feedback

Blogs

Reflexion — Lilian Weng's Agent Survey