v9.1.1
Menu
Sign In

Learn by Interaction

Reinforcement Learning Study Plan

An 8-chapter journey from the foundations of sequential decision-making to modern deep RL.

8

chapters

40

topics

4

free chapters

PathChapters8Practice1
01

The path

4 phases, in the order we recommend.

  1. 1

    Foundations

    Weeks 1-3
    Ch 1: Introduction to RLCh 2: Markov Decision ProcessesCh 3: Dynamic Programming

    The vocabulary, the MDP formalism, and planning with a known model

  2. 2

    Model-Free Learning

    Weeks 4-5
    Ch 4: Monte Carlo & TDCh 5: Q-Learning & SARSA

    Learning value functions directly from experience

  3. 3

    Deep RL

    Week 6
    Ch 6: Deep Q-Networks (DQN)

    Function approximation, experience replay, target networks

  4. 4

    Policy Optimization

    Weeks 7-8
    Ch 7: Policy GradientsCh 8: Actor-Critic & PPO

    Optimizing policies directly, up to modern PPO

Tip: RL builds on probability and gradients — the Foundations and ML plans are useful prerequisites.

Pro chapters (4-8) require a premium subscription.

02

Chapters

4 of 8 chapters free. The rest unlock with Pro.

CH 01

Introduction to Reinforcement Learning

Learning from interaction, the agent-environment loop, rewards and returns, policies and value functions, and the exploration-exploitation tradeoff.

5 topicsStart→
CH 02

Markov Decision Processes

Formalizing sequential decisions: states, actions, transition dynamics, the Markov property, return, and the Bellman equations.

5 topicsStart→
CH 03

Dynamic Programming

Solving known MDPs with planning: policy evaluation, policy improvement, policy iteration, value iteration, and generalized policy iteration.

5 topicsStart→
CH 04

Monte Carlo & Temporal-Difference Learning

Learning from experience without a model: Monte Carlo prediction, TD(0), bootstrapping, and the bias-variance tradeoff between them.

5 topicsStart→
CH 05

Q-Learning & SARSA

PRO

Model-free control with action-value methods: SARSA (on-policy), Q-learning (off-policy), epsilon-greedy exploration, and convergence.

5 topicsUnlock with Pro
CH 06

Deep Q-Networks (DQN)

PRO

Scaling RL with function approximation: neural network value functions, experience replay, target networks, and DQN improvements.

5 topicsUnlock with Pro
CH 07

Policy Gradient Methods

PRO

Optimizing policies directly: the policy gradient theorem, REINFORCE, baselines and variance reduction, and continuous action spaces.

5 topicsUnlock with Pro
CH 08

Actor-Critic & PPO

PRO

Combining value and policy learning: actor-critic architectures, advantage estimation (GAE), A2C, and Proximal Policy Optimization.

5 topicsUnlock with Pro
03

Practice

Problem sets that drill this plan's topics.

RL 1: Foundations

Practice the maths behind every RL algorithm in the plan.

Returns, Policies, and Value FunctionsMarkov Decision Processes and Bellman EquationsDynamic Programming+2
5 topicsView problems→

Curriculum inspired by Sutton & Barto's Reinforcement Learning: An Introduction, taking you from RL foundations to modern deep reinforcement learning.