Expected Immediate Reward of a Policy
Problem Statement
For a single state, the expected immediate reward under a policy is:
rπ(s)=∑a​π(a∣s)r(s,a)
Given the policy pi and the per-action expected rewards r (aligned by action), implement expected_reward(pi, r).
Example:
expected_reward([0.5, 0.5], [1.0, 3.0])
2.0
- Identify the policy probabilities and corresponding rewards from the input: the policy is π=[0.5,0.5] and the rewards are r=[1.0,3.0].
- Calculate the contribution of the first action by multiplying its probability by its reward: 0.5×1.0=0.5.
- Calculate the contribution of the second action by multiplying its probability by its reward: 0.5×3.0=1.5.
- Sum these individual contributions to find the total expected immediate reward: 0.5+1.5=2.0.
- The final output is 2.0
Constraints:
len(pi) == len(r), valid distribution.- Return a float.
1. Background Knowledge
In a Markov Decision Process (MDP), an agent interacts with an environment by selecting actions based on its current state. A policy π defines the agent's behavior: for each state s, it specifies a probability distribution over actions. Formally, π(a∣s) is the probability of taking action a in state s, and these probabilities must sum to 1 for all actions in that state.
The expected immediate reward rπ(s) represents the average reward the agent can expect to receive in state s when following policy π. It is computed as a weighted sum of the rewards for each action, where the weights are the policy's action probabilities. This concept is foundational to Bellman equations, which relate the value of a state to the expected rewards and future values of successor states. Understanding this simple expectation is the first step toward computing state values, action values, and ultimately optimal policies.
2. Algorithm Approach
This problem is a straightforward application of the dot product between two vectors: the policy vector (probabilities) and the reward vector (expected rewards per action). The algorithm pattern is:
- Validate that the policy is a valid probability distribution (non-negative, sums to 1).
- Compute the element-wise product of the policy and reward vectors.
- Sum the resulting products to obtain the expected reward.
This is essentially computing ∑a​π(a∣s)⋅r(s,a), which is the definition of the expected value of a discrete random variable.
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.