Converting Between V and Q

Problem Statement

Implement the two identities that connect the state-value function $v_\pi$ and the action-value function $q_\pi$.

Background

There are two value functions and they carry the same information in different shapes.

$v_\pi(s)$ is the expected return from state s when the policy chooses the action, so it is the policy-weighted average of the action values:

$v_\pi(s) = \sum_a \pi(a \mid s)\, q_\pi(s, a)$

$q_\pi(s, a)$ is the expected return from state s when you commit to action a first and follow the policy afterwards. Because the action is fixed, you pay the expected immediate reward and then land in a successor state whose value is already $v_\pi$:

$q_\pi(s, a) = r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, v_\pi(s')$

Note the asymmetry: going from q to v needs only the policy, while going from v to q needs the model (r and p). That asymmetry is exactly why model-free control learns Q and not V — with Q in hand you can act greedily without knowing the dynamics at all.

Your Task

Implement both:

def v_from_q(pi, q):
    ...

def q_from_v(P, R, v, gamma):
    ...

• pi[s][a] — probability of taking action a in state s.
• q[s][a] — action value.
• P[s][a][s2] — probability of moving to state s2 from s under action a.
• R[s][a] — expected immediate reward r(s, a).
• v[s] — state value.
• gamma — discount factor.

v_from_q returns a list of n_states floats. q_from_v returns a list of n_states lists of n_actions floats.

Input / Output Format

All inputs are nested Python lists of floats. Outputs are nested lists of floats, rounded to 4 decimals by the grader.

Sample

pi = [[0.6, 0.4], [0.2, 0.8], [1.0, 0.0]]
q  = [[1.0, 2.0], [3.0, -1.0], [0.5, 0.5]]
print([round(x, 4) for x in v_from_q(pi, q)])

Output:

[1.4, -0.2, 0.5]

State 0: 0.61.0 + 0.42.0 = 1.4. State 1: 0.23.0 + 0.8(-1.0) = -0.2.