Chapter 5: Q-Learning & SARSA
Model-free control with action values: SARSA as on-policy TD control, Q-learning as its off-policy counterpart, what the on-policy/off-policy distinction actually costs, and the maximization-bias fixes of Expected SARSA and Double Q-learning.
Chapter Overview
Chapter 4 built the machinery for learning values from experience. This chapter turns it into control. The move is small — learn instead of so that improvement needs no model — but it produces the two algorithms that defined reinforcement learning for two decades.
SARSA and Q-learning differ by a single term in the update, and that one difference is the on-policy/off-policy distinction. SARSA learns the value of the policy it is actually following, including its exploratory mistakes. Q-learning learns the value of the greedy policy regardless of what it is doing. On the cliff-walking problem they produce visibly different behaviour: one takes the safe route, the other the optimal-but-risky one. Understanding why is understanding the distinction that governs everything from replay buffers to importance sampling.
Chapter Roadmap
Click any topic to jump in
Action-Value Methods
Learning Q rather than V makes greedy improvement possible without any model of the environment.
Use the action you will actually take, or the best action available — one choice, two algorithms.
SARSA
On-policy TD control: bootstrap from the action actually taken next, so exploration is priced into the value.
Q-Learning
Off-policy TD control: bootstrap from the best next action, learning the greedy policy while behaving otherwise.
On vs Off-Policy
Whether the policy being evaluated is the one generating the data — the distinction that decides if replay is legal.
Expected SARSA & Double Q
Reduce update variance by averaging over actions, and remove maximization bias with two decoupled estimators.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.