Chapter 6: Deep Q-Networks (DQN)
Scale action-value learning to large state spaces with neural networks: function approximation and the deadly triad, the DQN architecture, experience replay, target networks, and the Double, Dueling and Prioritized improvements that make it work.
- Topics
- 5
- Videos
- 3
Chapter Overview
Everything so far assumed a table with one entry per state-action pair. That assumption breaks immediately on real problems: an Atari screen has more configurations than there are atoms in the observable universe, so a table can neither be stored nor filled. Function approximation replaces the table with a parameterized function — here, a neural network — that generalizes across states it has never seen.
The trouble is that the convergence guarantees from Chapters 4 and 5 do not survive the substitution. Combining function approximation, bootstrapping and off-policy updates — the 'deadly triad' — can make learning diverge outright. DQN's contribution was not the neural network, which had been tried before, but two engineering ideas that tame the instability: replay a large buffer of past transitions in random order, and hold the bootstrap target fixed for thousands of steps. Nearly every later value-based method is a refinement of that recipe.
Chapter Roadmap
Click any topic to jump in
Function Approximation
Replace the lookup table with a parameterized function so values generalize across similar states — and lose the tabular guarantees.
DQN Architecture
A convolutional network mapping raw pixels to one Q-value per action, trained by regression on TD targets.
With training no longer diverging, the remaining errors are worth attacking one at a time.
Experience Replay
Store transitions and sample them randomly, breaking temporal correlation and reusing each one many times.
Target Networks
Freeze a copy of the network to supply bootstrap targets, so the regression target stops moving every step.
Double, Dueling & PER
Fix maximization bias, separate state value from action advantage, and sample the most informative transitions more often.
Function Approximation
The DQN Architecture
Experience Replay
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.