Loading...
RL 1: Foundations — all 50 problems
Fifteen implementation exercises covering the computational core of reinforcement learning: discounted returns, the Bellman expectation and optimality equations, dynamic programming, temporal-difference learning and control, and modern policy-gradient objectives including GAE and PPO's clipped surrogate.
Dynamic Programming
- Greedy Policy ImprovementEasy
- Greedy Policy from Action ValuesEasy
- Policy Stability CheckEasy
- Iterative Policy EvaluationMedium
- One Sweep of Iterative Policy EvaluationMedium
- Q-Values from State ValuesMedium
- Value Iteration Bellman ErrorMedium
- Async In-Place Value Iteration SweepMedium
- Value Iteration on a Small MDPHard
- Full Value Iteration to ConvergenceHard
Markov Decision Processes and Bellman Equations
- Marginalising the MDP Dynamics FunctionEasy
- Expected Immediate Reward of a PolicyEasy
- Normalize a Transition RowEasy
- Solve the Bellman Expectation Equation ExactlyMedium
- Bellman Optimality Backup on Action ValuesMedium
- One-Step Bellman Expectation BackupMedium
- Bellman Optimality BackupMedium
- Marginalize Next-State DistributionMedium
- Effective Horizon of a Discount FactorMedium
- Policy Evaluation by Linear SolveHard
Policy Gradients and Advantage Estimation
- Log-Probability of a Softmax PolicyEasy
- Entropy of a PolicyEasy
- REINFORCE Gradient of a Softmax PolicyMedium
- Generalized Advantage EstimationMedium
- REINFORCE Loss TermMedium
- Return Standardization BaselineMedium
- TD Residuals for GAEMedium
- PPO Clipped Objective (Single Sample)Medium
- PPO Clipped Surrogate ObjectiveHard
- GAE Over a Full TrajectoryHard
Returns, Policies, and Value Functions
- Discounted Return of a TrajectoryEasy
- Epsilon-Greedy Action DistributionEasy
- Undiscounted Finite-Horizon ReturnEasy
- Greedy Action from Action ValuesEasy
- Softmax Policy ProbabilitiesEasy
- Converting Between V and QMedium
- State Value from Action ValuesMedium
- Reward-to-Go for Every TimestepMedium
- Advantage from Q and VMedium
- Return Variance Under Stochastic RewardsHard
Temporal-Difference Learning and Control
- TD(0) Prediction Over a Transition StreamEasy
- TD(0) TargetEasy
- TD ErrorEasy
- TD(0) Value Update StepEasy
- SARSA, Q-Learning and Expected SARSA TargetsMedium
- TD(0) Over a Transition StreamMedium
- SARSA vs Q-Learning TargetsMedium
- Expected SARSA TargetMedium
- n-step Returns and the Lambda ReturnHard
- n-step and Lambda ReturnsHard