TD Error

easyPROTemporal-Difference Learning and ControlReinforcement LearningRL 1: Foundations

Problem Statement

The TD error (delta) drives the value update:

$\delta = r + \gamma V(s') - V(s)$

Implement td_error(reward, gamma, v_next, v_current) returning a float.

Example

Input

td_error(1.0, 0.9, 10.0, 5.0)

Output

5.0

Reasoning

Identify the input parameters for the TD error formula: immediate reward $r = 1.0$ , discount factor $\gamma = 0.9$ , next state value $V(s') = 10.0$ , and current state value $V(s) = 5.0$ .
Calculate the discounted value of the next state to account for future rewards: $\gamma \cdot V(s') = 0.9 \times 10.0 = 9.0$ .
Add the immediate reward to the discounted next state value to determine the target value: $r + \gamma \cdot V(s') = 1.0 + 9.0 = 10.0$ .
Subtract the current state value from the target value to find the difference (error): $10.0 - 5.0 = 5.0$ .
The final output is 5.0

Constraints

Return a float.

Learn this in

Introduction to Reinforcement LearningReinforcement Learning Monte Carlo & Temporal-Difference LearningReinforcement Learning Actor-Critic & PPOReinforcement Learning

More from RL 1: Foundations

GAE Over a Full Trajectoryhard PPO Clipped Objective (Single Sample)medium TD Residuals for GAEmedium Return Standardization Baselinemedium REINFORCE Loss Termmedium

Concept primerPREVIEW

1. Background Knowledge

Temporal-Difference (TD) learning is a family of reinforcement learning algorithms that update value estimates based on the difference between two successive predictions. The core quantity is the TD error (often denoted $\delta$ ), which measures how much the observed outcome deviates from what the current value function predicted. In its simplest form for a state-value function $V(s)$ , the TD error is defined as:

\delta = r + \gamma V(s') - V(s)

where $r$ is the immediate reward received, $\gamma \in [0, 1]$ is the discount factor, $V(s')$ is the estimated value of the next state, and $V(s)$ is the estimated value of the current state. This formula blends the immediate reward with the discounted future value, then subtracts the current estimate to produce a correction signal.

The TD error serves as the learning signal in algorithms like TD(0), SARSA, and Q-learning. When $\delta > 0$ , the environment was better than expected, so the value estimate should increase. When $\delta < 0$ , the environment was worse than expected, so the value estimate should decrease. When $\delta = 0$ , the prediction was perfectly accurate. The discount factor $\gamma$ controls how much future rewards matter relative to immediate ones; a value of $\gamma = 0$ makes the agent myopic, while $\gamma$ close to $1$ makes it far-sighted.

2. Algorithm Approach

This problem is a direct implementation of a mathematical formula. The approach is straightforward:

Accept four scalar inputs: reward, gamma, v_next, and v_current.
Compute the target value as the sum of the immediate reward and the discounted next-state value: $r + \gamma \cdot V(s')$ .
Subtract the current value estimate $V(s)$ from this target.
Return the result as a float.

There is no loop, no data structure, and no branching logic. The entire computation is a single arithmetic expression. The key is to correctly map each variable to its role in the TD error formula.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results