Chapter 8: Actor-Critic & PPO
Combine value and policy learning: the actor-critic architecture, generalized advantage estimation, synchronous and asynchronous A2C/A3C, the trust-region idea behind TRPO, and the clipped surrogate objective that made PPO the default algorithm.
Chapter Overview
Chapter 7 ended with a working but impractical method: REINFORCE with a baseline is unbiased and general, yet needs complete episodes and still has high variance. Chapter 4 already supplied the fix — bootstrap. Replacing the sampled return with a learned value estimate gives the actor-critic architecture, where a policy (the actor) is improved using advantages supplied by a value function (the critic).
That combination is powerful but delicate. Policy gradient steps are only valid locally: the gradient is computed under the current policy's trajectory distribution, so a step large enough to change that distribution invalidates the very data it was computed from. TRPO addressed this with a hard constraint on how far the policy may move, at considerable implementation cost. PPO achieved nearly the same effect by clipping a probability ratio — a change of a few lines that turned out to be robust enough to become the default algorithm across robotics, games, and RLHF for language models.
Chapter Roadmap
Click any topic to jump in
Actor-Critic
A policy improved by advantages from a learned value function — bootstrapping brings policy gradients online.
GAE
Exponentially weight n-step advantage estimates to tune the bias-variance tradeoff with a single parameter.
A hard KL constraint works but demands second-order optimization.
A2C / A3C
Run many environments at once so a batch of decorrelated transitions replaces the replay buffer.
Trust Regions & TRPO
Constrain the KL divergence between successive policies so each update stays in the region where the estimate is valid.
PPO
Approximate the trust region by clipping the probability ratio — simple, robust, and the field's default.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.