TD(0) Value Update Step

easyPROTemporal-Difference Learning and ControlReinforcement LearningRL 1: Foundations

Problem Statement

Apply one TD(0) update to the value of the current state:

$V(s) \leftarrow V(s) + \alpha\, [\, r + \gamma V(s') - V(s)\,]$

Implement td_update(v_current, reward, gamma, v_next, alpha) returning the new V(s).

Example

Input

td_update(5.0, 1.0, 0.9, 10.0, 0.1)

Output

5.5

Reasoning

Calculate the temporal difference error (TD error) by combining the immediate reward, the discounted value of the next state, and the current value estimate: $\delta = 1.0 + (0.9 \times 10.0) - 5.0 = 1.0 + 9.0 - 5.0 = 5.0$ .
Scale the TD error by the learning rate to determine the magnitude of the adjustment: $\alpha \times \delta = 0.1 \times 5.0 = 0.5$ .
Update the current value estimate by adding this adjustment to the original value: $V(s)_{new} = 5.0 + 0.5 = 5.5$ .
The final output is 5.5

Constraints

0 < alpha <= 1.
Return a float.

Learn this in

Introduction to Reinforcement LearningReinforcement Learning Monte Carlo & Temporal-Difference LearningReinforcement Learning

More from RL 1: Foundations

GAE Over a Full Trajectoryhard PPO Clipped Objective (Single Sample)medium TD Residuals for GAEmedium Return Standardization Baselinemedium REINFORCE Loss Termmedium

Concept primerPREVIEW

1. Background Knowledge

Temporal-Difference (TD) learning is a family of reinforcement learning methods that update value estimates by blending the current estimate with a new observation. Unlike Monte Carlo methods, which wait for an episode to finish before updating, TD methods update after every single transition. This makes them more sample-efficient and capable of handling infinite-horizon tasks where episodes never terminate.

The core idea behind TD(0) is the TD error, defined as the difference between the observed one-step return and the current value estimate. For a transition from state $s$ to state $s'$ with reward $r$ , the one-step return is $r + \gamma V(s')$ , where $\gamma$ is the discount factor. The TD error is:

$\delta = r + \gamma V(s') - V(s)$

A positive TD error means the observed outcome was better than expected, so the value estimate should increase. A negative TD error means the outcome was worse than expected, so the estimate should decrease. The update rule moves the current estimate toward the target by a fraction $\alpha$ (the learning rate) of the TD error.

In this problem, you are implementing exactly one such update step. The function takes the current value estimate, the observed reward, the discount factor, the next state's value estimate, and the learning rate, then returns the updated value.

2. Algorithm Approach

The approach is a direct application of the TD(0) update formula. There is no loop, no data structure, and no branching logic beyond the arithmetic itself. The algorithm consists of:

Compute the TD target: $r + \gamma \cdot V(s')$
Compute the TD error: $\text{target} - V(s)$
Apply the update: $V(s) \leftarrow V(s) + \alpha \cdot \text{error}$

This is a single scalar computation. The key conceptual step is recognizing that the update is a weighted average between the old estimate and the target:

$V_{\text{new}} = (1 - \alpha)\, V(s) + \alpha\, (r + \gamma\, V(s'))$

Both forms are algebraically equivalent. The first form (error-based) is more common in RL literature and makes the direction of the update intuitive.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results