📘

Undiscounted Finite-Horizon Return

EasyReinforcement Learning 📚 RL 1: Foundations

Problem Statement

Compute the undiscounted return of an episode: the plain sum of all rewards collected, i.e. the discounted return with gamma = 1.

$G_0 = \sum_{t=1}^{T} R_t$

Implement undiscounted_return(rewards) returning a Python float.

Example:

Input:

undiscounted_return([1.0, 2.0, 3.0])

Output:

6.0

Reasoning:

Identify the sequence of rewards collected during the episode from the input: $R_1 = 1.0$ , $R_2 = 2.0$ , and $R_3 = 3.0$ .
Compute the undiscounted return by summing these rewards, as the discount factor $\gamma = 1$ implies no decay in value over time: $G_0 = 1.0 + 2.0 + 3.0$ .
Perform the arithmetic addition to find the total accumulated reward: $G_0 = 6.0$ .
The final output is 6.0

Constraints:

0 <= len(rewards) <= 10000
Rewards may be negative or fractional.
An empty reward list returns 0.0.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

In reinforcement learning, an episode is a sequence of interactions between an agent and an environment, starting from some initial state and ending at a terminal state. At each time step $t$ , the agent receives a scalar reward $R_t$ that signals how good the previous action was. The cumulative performance over the entire episode is quantified by the return, which aggregates these rewards into a single scalar value.

The most general form of the return is the discounted return, defined as $G_0 = \sum_{t=1}^{T} \gamma^{t-1} R_t$ , where $\gamma \in [0, 1]$ is the discount factor. The discount factor controls how much the agent values future rewards relative to immediate ones. When $\gamma = 1$ , the discounting effect vanishes entirely, and the return becomes the simple arithmetic sum of all rewards collected during the episode. This is known as the undiscounted return or finite-horizon return, and it is appropriate when the episode has a known, finite length $T$ and there is no reason to devalue later rewards.

Understanding the undiscounted return is foundational because it is the simplest case of the return definition. It appears in finite-horizon problems, in the analysis of episodic tasks with fixed length, and as a baseline against which discounted returns are compared. In practice, many RL algorithms (e.g., Monte Carlo methods, temporal-difference learning) compute or estimate returns, and the undiscounted case serves as a useful sanity check and pedagogical stepping stone.

2. Algorithm Approach

This problem is a straightforward linear scan (also called a reduction or fold) over a sequence of numbers. The algorithmic pattern is:

Initialize an accumulator variable to zero.
Iterate through each reward in the input list.
Add each reward to the accumulator.
Return the final accumulated sum.

This is equivalent to computing the prefix sum of the entire array, or applying the built-in sum() function in Python. No sorting, searching, or dynamic programming is required. The key insight is that the undiscounted return is simply the arithmetic mean multiplied by the number of steps, or more directly, the total sum of the reward sequence.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.