📘

TD(0) Value Update Step

EasyReinforcement Learning 📚 RL 1: Foundations

Problem Statement

Apply one TD(0) update to the value of the current state:

$V(s) \leftarrow V(s) + \alpha\, [\, r + \gamma V(s') - V(s)\,]$

Implement td_update(v_current, reward, gamma, v_next, alpha) returning the new V(s).

Example:

Input:

td_update(5.0, 1.0, 0.9, 10.0, 0.1)

Output:

5.5

Reasoning:

Calculate the temporal difference error (TD error) by combining the immediate reward, the discounted value of the next state, and the current value estimate: $\delta = 1.0 + (0.9 \times 10.0) - 5.0 = 1.0 + 9.0 - 5.0 = 5.0$ .
Scale the TD error by the learning rate to determine the magnitude of the adjustment: $\alpha \times \delta = 0.1 \times 5.0 = 0.5$ .
Update the current value estimate by adding this adjustment to the original value: $V(s)_{new} = 5.0 + 0.5 = 5.5$ .
The final output is 5.5

Constraints:

0 < alpha <= 1.
Return a float.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

Temporal-Difference (TD) learning is a family of reinforcement learning methods that update value estimates by blending the current estimate with a new observation. Unlike Monte Carlo methods, which wait for an episode to finish before updating, TD methods update after every single transition. This makes them more sample-efficient and capable of handling infinite-horizon tasks where episodes never terminate.

The core idea behind TD(0) is the TD error, defined as the difference between the observed one-step return and the current value estimate. For a transition from state $s$ to state $s'$ with reward $r$ , the one-step return is $r + \gamma V(s')$ , where $\gamma$ is the discount factor. The TD error is:

$\delta = r + \gamma V(s') - V(s)$

A positive TD error means the observed outcome was better than expected, so the value estimate should increase. A negative TD error means the outcome was worse than expected, so the estimate should decrease. The update rule moves the current estimate toward the target by a fraction $\alpha$ (the learning rate) of the TD error.

In this problem, you are implementing exactly one such update step. The function takes the current value estimate, the observed reward, the discount factor, the next state's value estimate, and the learning rate, then returns the updated value.

2. Algorithm Approach

The approach is a direct application of the TD(0) update formula. There is no loop, no data structure, and no branching logic beyond the arithmetic itself. The algorithm consists of:

Compute the TD target: $r + \gamma \cdot V(s')$
Compute the TD error: $\text{target} - V(s)$
Apply the update: $V(s) \leftarrow V(s) + \alpha \cdot \text{error}$

This is a single scalar computation. The key conceptual step is recognizing that the update is a weighted average between the old estimate and the target:

$V_{\text{new}} = (1 - \alpha)\, V(s) + \alpha\, (r + \gamma\, V(s'))$

Both forms are algebraically equivalent. The first form (error-based) is more common in RL literature and makes the direction of the update intuitive.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.