Chapter 4: Monte Carlo & Temporal-Difference Learning
Drop the model and learn from experience: Monte Carlo prediction and control, TD(0), the bias-variance tradeoff between sampling and bootstrapping, and the n-step and TD(lambda) methods that interpolate between them.
Chapter Overview
Dynamic programming needed the transition dynamics. Almost no interesting problem provides them. This chapter removes that assumption and replaces the expectation in the Bellman update with something the agent can actually obtain: sampled experience.
There are two distinct ways to do this, and the difference between them is the central idea of the chapter. Monte Carlo methods wait until an episode ends and use the actual return — unbiased, but high variance and unusable until termination. Temporal-difference methods use a one-step reward plus the current estimate of the next state's value — biased, because the estimate is wrong early on, but low variance and updatable after every single step. Neither dominates. n-step methods and TD(lambda) span the space between them, and understanding that spectrum explains design choices in every algorithm that follows.
Chapter Roadmap
Click any topic to jump in
Monte Carlo Prediction
Estimate a state's value by averaging the actual returns observed after visiting it — no model, no bootstrapping.
Monte Carlo Control
Plug sampled action-value estimates into generalized policy iteration, with exploration supplied by epsilon-greedy.
One-step TD and full-return Monte Carlo are the endpoints, not rival families.
TD(0) Prediction
Update after every step using the observed reward plus the discounted estimate of the next state.
Bootstrapping vs Sampling
The bias-variance axis that separates TD from Monte Carlo and explains when each one wins.
n-step & TD(lambda)
Interpolate between the two extremes by looking n steps ahead, or by averaging all n at once with eligibility traces.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.