TD Error

1. Background Knowledge

Temporal-Difference (TD) learning is a family of reinforcement learning algorithms that update value estimates based on the difference between two successive predictions. The core quantity is the TD error (often denoted $\delta$), which measures how much the observed outcome deviates from what the current value function predicted. In its simplest form for a state-value function $V(s)$, the TD error is defined as:

where $r$ is the immediate reward received, $\gamma \in [0, 1]$ is the discount factor, $V(s')$ is the estimated value of the next state, and $V(s)$ is the estimated value of the current state. This formula blends the immediate reward with the discounted future value, then subtracts the current estimate to produce a correction signal.

The TD error serves as the learning signal in algorithms like TD(0), SARSA, and Q-learning. When $\delta > 0$, the environment was better than expected, so the value estimate should increase. When $\delta < 0$, the environment was worse than expected, so the value estimate should decrease. When $\delta = 0$, the prediction was perfectly accurate. The discount factor $\gamma$ controls how much future rewards matter relative to immediate ones; a value of $\gamma = 0$ makes the agent myopic, while $\gamma$ close to $1$ makes it far-sighted.

2. Algorithm Approach

This problem is a direct implementation of a mathematical formula. The approach is straightforward:

• Accept four scalar inputs: reward, gamma, v_next, and v_current.
• Compute the target value as the sum of the immediate reward and the discounted next-state value: $r + \gamma \cdot V(s')$.
• Subtract the current value estimate $V(s)$ from this target.
• Return the result as a float.

There is no loop, no data structure, and no branching logic. The entire computation is a single arithmetic expression. The key is to correctly map each variable to its role in the TD error formula.