easyPROTemporal-Difference Learning and ControlReinforcement LearningRL 1: Foundations→
Problem Statement
The TD(0) target for a single transition (s, r, s') is:
target=r+γV(s′)
Implement td_target(reward, gamma, v_next) returning a float. For a terminal transition callers pass v_next = 0.0.
Example
Input
td_target(1.0, 0.9, 10.0)
Output
10.0
Reasoning
Identify the input parameters for the TD(0) formula: the immediate reward r=1.0, the discount factor γ=0.9, and the value of the next state V(s′)=10.0.
Calculate the discounted future value by multiplying the discount factor by the next state's value: γ⋅V(s′)=0.9×10.0=9.0.
Add the immediate reward to the discounted future value to compute the target: r+9.0=1.0+9.0=10.0.
Temporal-Difference (TD) learning is a family of reinforcement learning methods that update value estimates using observed rewards and the value of the next state, rather than waiting for a full episode to finish. The core idea is bootstrapping: the estimate of a state's value is updated based on a combination of the immediate reward and the current estimate of the future. This makes TD methods more sample-efficient than Monte Carlo methods, which require complete return samples.
The TD(0) target is the one-step look-ahead estimate used in the simplest TD update. For a transition (s,r,s′), the target is defined as r+γV(s′), where γ∈[0,1] is the discount factor that controls how much future rewards are valued relative to immediate ones. When γ=0, the agent is myopic and only cares about the immediate reward; when γ approaches 1, future rewards are nearly as important as present ones.
A critical detail is the handling of terminal states. When the next state s′ is terminal, there are no further rewards to collect, so the value of the next state is defined as V(s′)=0. This effectively truncates the bootstrapping chain at the episode boundary. In practice, callers signal this by passing v_next = 0.0 for terminal transitions, which simplifies the target to just the immediate reward r.
2. Algorithm Approach
This problem follows a direct formula evaluation pattern. There is no iterative search, dynamic programming, or optimization involved. The approach is:
Accept the three inputs: reward, gamma, and v_next.
Compute the discounted value of the next state: γ⋅V(s′).
Add the immediate reward to this discounted value.
Return the result as a float.
The key conceptual step is recognizing that v_next already encodes the terminal-state convention. You do not need to check whether the state is terminal; the caller has already set v_next = 0.0 in that case. Your function simply applies the arithmetic formula uniformly.
3. Step-by-Step Strategy
PRO
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
PRO
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.