📘

SARSA vs Q-Learning Targets

MediumReinforcement Learning 📚 RL 1: Foundations

Problem Statement

Compute both control targets for a transition, given the next-state action values q_next (a list over actions), the on-policy next action index a_next, reward r and discount gamma:

SARSA (on-policy): target = r + gamma * q_next[a_next]
Q-Learning (off-policy): target = r + gamma * max(q_next)

Implement control_targets(reward, gamma, q_next, a_next) returning the tuple (sarsa, qlearning).

Example:

Input:

control_targets(1.0, 0.9, [2.0, 5.0], 0)

Output:

(2.8, 5.5)

Reasoning:

Identify the input parameters: reward $r = 1.0$ , discount factor $\gamma = 0.9$ , next-state action values $q_{\text{next}} = [2.0, 5.0]$ , and the on-policy action index $a_{\text{next}} = 0$ .
Compute the SARSA target by using the value of the specific action taken in the next state (index 0): $sarsa = 1.0 + 0.9 \times 2.0 = 1.0 + 1.8 = 2.8$ .
Compute the Q-Learning target by using the maximum value among all possible next actions to represent the optimal policy: $q_{\text{learning}} = 1.0 + 0.9 \times \max(2.0, 5.0) = 1.0 + 0.9 \times 5.0 = 1.0 + 4.5 = 5.5$ .
The final output is (2.8, 5.5)

Constraints:

0 <= a_next < len(q_next).
Return a tuple of two floats.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

Temporal-Difference (TD) learning bridges dynamic programming and Monte Carlo methods by bootstrapping value estimates from other value estimates. In control problems, we learn an action-value function $Q(s, a)$ that estimates the expected return of taking action $a$ in state $s$ and following a policy thereafter. The core update rule adjusts $Q$ toward a TD target, which is a one-step lookahead estimate of the return: the immediate reward plus a discounted estimate of future value.

The critical distinction between SARSA and Q-Learning lies in which next-state value is used in the target. SARSA is an on-policy algorithm: it learns the value of the policy it is actually following. Its target uses the specific action $a_{next}$ that the agent will actually take in the next state. Q-Learning is an off-policy algorithm: it learns the value of the optimal greedy policy regardless of the behavior policy. Its target uses $\max_a Q(s', a)$ , representing the best possible action in the next state. This single difference—using $Q(s', a_{next})$ versus $\max_a Q(s', a)$ —determines whether the algorithm is on-policy or off-policy.

The discount factor $\gamma \in [0, 1]$ controls how much future rewards are valued relative to immediate rewards. When $\gamma = 0$ , the agent is myopic and only cares about the immediate reward. When $\gamma$ is close to 1, the agent is far-sighted and heavily weights long-term returns. The TD target formula $r + \gamma \cdot V_{next}$ is the fundamental building block of all TD-based RL algorithms.

2. Algorithm Approach

This problem is a direct application of the two TD target formulas. The approach is straightforward:

Compute the SARSA target by indexing into q_next at position a_next and combining with reward and discount.
Compute the Q-Learning target by finding the maximum value in q_next and combining with reward and discount.
Return both as a tuple.

No iterative learning loop is needed—this is a single-step target computation, which is the atomic operation that would be repeated inside the actual SARSA or Q-Learning update rule.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.