Chapter 7: Policy Gradient Methods
Optimize the policy directly instead of deriving it from values: why value-based methods fail on continuous actions, the policy gradient theorem, REINFORCE, baselines and variance reduction, and Gaussian policies for continuous control.
Chapter Overview
Every method so far learned a value function and derived a policy from it by maximization. That indirection has three costs. It cannot handle continuous actions, because the arg-max becomes an optimization problem at every step. It cannot represent a genuinely stochastic optimum, which is required whenever the environment is partially observable or adversarial. And small changes in value estimates can flip the arg-max, making the policy change discontinuously.
Policy gradient methods remove the indirection: parameterize the policy itself and ascend the gradient of expected return. The obstacle is that the objective depends on the distribution of trajectories, which depends on the policy in a way that seems to require differentiating through the environment. The policy gradient theorem shows that it does not — the gradient can be written as an expectation over trajectories the agent can simply sample. That result is what makes the whole family possible, and it leads directly to the actor-critic and PPO algorithms of Chapter 8.
Chapter Roadmap
Click any topic to jump in
Why Policy Gradients
Continuous actions, genuinely stochastic optima, and smooth policy changes are all out of reach for arg-max methods.
Policy Gradient Theorem
The gradient of expected return is an expectation you can sample — no derivative of the environment required.
Baselines and causality reduce variance by orders of magnitude at no cost in bias.
REINFORCE
The theorem turned into an algorithm: weight the log-probability gradient of each action by the return that followed.
Baselines & Variance
Subtracting a state-dependent baseline cuts variance dramatically without introducing bias.
Continuous Actions
Output the parameters of a distribution rather than a choice — the setting where policy gradients are not merely better but necessary.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.