📘

Effective Horizon of a Discount Factor

MediumReinforcement Learning 📚 RL 1: Foundations

Problem Statement

The effective planning horizon implied by a discount factor is often summarized as 1 / (1 - gamma) — the sum of the geometric discount series sum_{t=0}^inf gamma^t. For gamma == 1 the horizon is infinite; represent that with float('inf').

Implement effective_horizon(gamma) returning a float.

Example:

Input:

effective_horizon(0.9)

Output:

10.0

Reasoning:

Check if the discount factor $\gamma = 0.9$ is greater than or equal to $1.0$ to determine if the horizon is infinite; since $0.9 < 1.0$ , the horizon is finite.
Calculate the denominator of the geometric series sum by subtracting the discount factor from $1$ : $1.0 - 0.9 = 0.1$ .
Compute the effective horizon by dividing $1$ by this denominator: $1.0 / 0.1 = 10.0$ .
The final output is 10.0

Constraints:

0.0 <= gamma <= 1.0
gamma == 1.0 returns float('inf').
Otherwise return 1 / (1 - gamma).

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

In Markov Decision Processes (MDPs), the discount factor $\gamma \in [0, 1]$ controls how much an agent values future rewards relative to immediate ones. The total expected return from time $t=0$ onward is defined as:

G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k}

The term $\gamma^k$ acts as a weight that decays exponentially. When $\gamma$ is close to 1, the agent is "patient" and considers distant rewards almost as important as near-term ones. When $\gamma$ is small, the agent is "myopic" and mostly cares about immediate outcomes.

The effective horizon is a heuristic measure of how many steps into the future the agent effectively "sees." It is derived from the sum of the pure discount weights (ignoring actual reward magnitudes):

H = \sum_{k=0}^{\infty} \gamma^k

This is a geometric series. For $|\gamma| < 1$ , the series converges to a finite value. For $\gamma = 1$ , every term equals 1, so the sum diverges to infinity. This distinction is critical in reinforcement learning: a finite horizon implies the agent's planning is bounded, while an infinite horizon implies the agent must consider all future consequences equally.

2. Algorithm Approach

This problem is a direct application of the closed-form solution for a geometric series. Rather than summing terms iteratively (which would be inefficient and prone to floating-point accumulation errors), you should use the analytical formula:

\sum_{k=0}^{\infty} r^k = \frac{1}{1 - r} \quad \text{for } |r| < 1

The algorithmic pattern here is:

Check for the special case where the series diverges ( $\gamma = 1$ ).
Apply the closed-form formula for all other valid inputs.

This is a constant-time mathematical evaluation, not an iterative computation.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.