Expected SARSA Target

mediumPROTemporal-Difference Learning and ControlReinforcement LearningRL 1: Foundations

Problem Statement

Expected SARSA replaces the sampled next action with the expectation under the next-state policy:

$target = r + \gamma \sum_a \pi(a\mid s')\, Q(s', a)$

Given reward r, discount gamma, next-state policy pi_next and action values q_next (aligned), implement expected_sarsa_target(reward, gamma, pi_next, q_next) returning a float.

Example

Input

expected_sarsa_target(1.0, 0.9, [0.5, 0.5], [2.0, 4.0])

Output

3.7

Reasoning

Calculate the expected action value for the next state by taking the dot product of the policy probabilities and the corresponding Q-values: $0.5 \times 2.0 + 0.5 \times 4.0 = 1.0 + 2.0 = 3.0$ .
Multiply this expected value by the discount factor $\gamma$ to account for the temporal distance of future rewards: $0.9 \times 3.0 = 2.7$ .
Add the immediate reward to the discounted expected future value to form the target: $1.0 + 2.7 = 3.7$ .
The final output is 3.7

Constraints

len(pi_next) == len(q_next), pi_next sums to 1.
Return a float.

Learn this in

Q-Learning & SARSAReinforcement Learning

More from RL 1: Foundations

GAE Over a Full Trajectoryhard PPO Clipped Objective (Single Sample)medium TD Residuals for GAEmedium Return Standardization Baselinemedium REINFORCE Loss Termmedium

Concept primerPREVIEW

1. Background Knowledge

In reinforcement learning, Temporal-Difference (TD) learning updates value estimates using observed rewards and the estimated value of the next state, rather than waiting for a full episode to finish. SARSA is an on-policy TD control algorithm that updates $Q(s, a)$ based on the actual next state $s'$ , next action $a'$ , and reward $r$ it experiences. The SARSA target is $r + \gamma Q(s', a')$ , where $a'$ is the action actually chosen by the current policy.

Expected SARSA is a variant that avoids the high variance introduced by sampling a single next action. Instead of using the specific $Q(s', a')$ for the sampled action, it computes the expectation of $Q(s', a)$ over all possible actions $a$ , weighted by the policy $\pi(a \mid s')$ . The target becomes:

\text{target} = r + \gamma \sum_{a} \pi(a \mid s') \, Q(s', a)

This is essentially a weighted average of the action values in the next state, where the weights are the probabilities assigned by the policy. Because it uses the full policy distribution, Expected SARSA has lower variance than standard SARSA but may converge more slowly in some settings. It remains on-policy because the expectation is taken under the same policy used for action selection.

2. Algorithm Approach

The core operation is a dot product between two vectors of equal length: the policy probabilities $\pi(a \mid s')$ and the action values $Q(s', a)$ . The algorithm pattern is:

Compute the weighted sum $\sum_a \pi(a \mid s') \, Q(s', a)$ , which is the expected action value in the next state.
Multiply this expectation by the discount factor $\gamma$ .
Add the immediate reward $r$ to obtain the TD target.

This is a straightforward vectorized reduction operation. No iterative loops over states or episodes are needed—just a single inner product over the action space.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results