Advantage from Q and V

mediumPROReturns, Policies, and Value FunctionsReinforcement LearningRL 1: Foundations

Problem Statement

The advantage of an action measures how much better it is than the state's average:

$A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$

Given action values q and the policy pi at a state, compute V = sum_a pi[a] q[a] and return the list of advantages A[a] = q[a] - V. Implement advantages(pi, q).

Example

Input

advantages([0.5, 0.5], [2.0, 4.0])

Output

[-1.0, 1.0]

Reasoning

Compute the state value $V$ by taking the weighted sum of the action values using the policy probabilities: $V = 0.5 \times 2.0 + 0.5 \times 4.0 = 1.0 + 2.0 = 3.0$ .
Calculate the advantage for the first action by subtracting the state value from its specific action value: $A[0] = 2.0 - 3.0 = -1.0$ .
Calculate the advantage for the second action by subtracting the state value from its specific action value: $A[1] = 4.0 - 3.0 = 1.0$ .
The final output is [-1.0, 1.0]

Constraints

len(pi) == len(q), valid distribution.
Under any valid policy, sum_a pi[a] * A[a] is 0.
Return a list of floats.

Learn this in

Reinforcement LearningML Study Plan Introduction to Reinforcement LearningReinforcement Learning Policy Gradient MethodsReinforcement Learning Actor-Critic & PPOReinforcement Learning

More from RL 1: Foundations

GAE Over a Full Trajectoryhard PPO Clipped Objective (Single Sample)medium TD Residuals for GAEmedium Return Standardization Baselinemedium REINFORCE Loss Termmedium

Concept primerPREVIEW

1. Background Knowledge

In reinforcement learning, the state-value function $V^\pi(s)$ represents the expected return when following policy $\pi$ from state $s$ . It is defined as the expectation of the action-value function $Q^\pi(s, a)$ over the actions chosen by the policy:

$V^\pi(s) = \sum_a \pi(a|s) Q^\pi(s, a)$

This is a weighted average of the action values, where the weights are the probabilities assigned by the policy. If the policy is deterministic (one action has probability 1), $V^\pi(s)$ simply equals the $Q$ -value of that action.

The advantage function $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ measures how much better (or worse) a specific action is compared to the average action available in that state. Actions with positive advantage are "better than average," while those with negative advantage are "worse than average." This concept is central to policy gradient methods like A2C and PPO, where the advantage is used to scale the gradient update, reducing variance compared to using raw returns.

2. Algorithm Approach

This is a straightforward expectation computation followed by an element-wise subtraction. The approach involves:

Compute the weighted sum of q using pi as weights to obtain $V$ .
Subtract $V$ from each element of q to produce the advantage vector.

No iterative or recursive logic is needed—this is a single-pass vectorized operation.

3. Step-by-Step Strategy

Validate inputs: Ensure pi and q have the same length. Optionally check that pi sums to 1 (within floating-point tolerance) and that all elements are non-negative.
Compute $V$ : Perform a dot product between pi and q. In NumPy, this is np.dot(pi, q) or np.sum(pi * q).
Compute advantages: Subtract the scalar $V$ from the array q element-wise: A = q - V.
Return the resulting list/array of advantages.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results