📘

State Value from Action Values

MediumReinforcement Learning 📚 RL 1: Foundations

Problem Statement

The state value under a policy is the policy-weighted average of action values:

$V^\pi(s) = \sum_a \pi(a\mid s)\, Q^\pi(s, a)$

Given the policy distribution pi and action values q (same length, aligned by action), implement state_value(pi, q).

Example:

Input:

state_value([0.5, 0.5], [2.0, 4.0])

Output:

3.0

Reasoning:

Identify the policy probabilities $\pi$ and action values $Q$ from the input: $\pi = [0.5, 0.5]$ and $Q = [2.0, 4.0]$ .
Calculate the weighted contribution for the first action by multiplying its probability by its value: $0.5 \times 2.0 = 1.0$ .
Calculate the weighted contribution for the second action similarly: $0.5 \times 4.0 = 2.0$ .
Sum these individual contributions to compute the policy-weighted average state value: $1.0 + 2.0 = 3.0$ .
The final output is 3.0

Constraints:

len(pi) == len(q), 1 <= len <= 1000
pi is a valid distribution (sums to 1).
Return a float.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

In Reinforcement Learning, the state value function $V^\pi(s)$ represents the expected return starting from state $s$ and following policy $\pi$ thereafter. The action value function $Q^\pi(s, a)$ represents the expected return starting from state $s$ , taking action $a$ , and then following $\pi$ . The relationship between them is fundamental: the state value is simply the expectation of the action value under the policy distribution.

Mathematically, if $\pi(a \mid s)$ is the probability of taking action $a$ in state $s$ , then:

V^\pi(s) = \sum_{a} \pi(a \mid s) \, Q^\pi(s, a)

This is a weighted average where the weights are the policy probabilities. Note that $\sum_a \pi(a \mid s) = 1$ for a valid probability distribution, so $V^\pi(s)$ lies within the range of the $Q^\pi(s, a)$ values. This concept is central to policy evaluation and policy improvement algorithms like Policy Iteration and Q-Learning.

2. Algorithm Approach

The problem reduces to computing a dot product between two vectors: the policy vector $\pi$ and the action value vector $Q$ . Since both vectors are aligned by action index, the $i$ -th element of $\pi$ corresponds to the $i$ -th element of $Q$ .

The general pattern is:

Verify that the inputs are valid (same length, non-negative probabilities, probabilities sum to 1).
Compute the element-wise product of $\pi$ and $Q$ .
Sum the resulting products.

This is a standard expectation calculation for a discrete random variable, where $Q$ is the variable and $\pi$ is its probability mass function.

3. Step-by-Step Strategy

Input Validation:

Check that pi and q have the same length.
Optionally, verify that pi is a valid probability distribution (all elements $\geq 0$ and $\sum \pi = 1$ ). This helps catch bugs early but may not be required by the problem.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.