📘

REINFORCE Loss Term

MediumReinforcement Learning 📚 RL 1: Foundations

Problem Statement

The REINFORCE objective maximizes sum_t log pi(a_t|s_t) * G_t. Implemented as a loss to minimize, it is the negative mean:

$L = -\frac{1}{T}\sum_{t=1}^{T} \log \pi(a_t\mid s_t)\, G_t$

Given aligned lists log_probs and returns, implement reinforce_loss(log_probs, returns) returning the scalar loss (float).

Example:

Input:

reinforce_loss([-0.7, -0.7], [1.0, 1.0])

Output:

0.7

Reasoning:

Identify the sequence length $T$ from the input lists, which is $2$ .
Compute the product of each log probability and its corresponding return: $(-0.7 \times 1.0) = -0.7$ and $(-0.7 \times 1.0) = -0.7$ .
Sum these products to find the total weighted log probability: $-0.7 + (-0.7) = -1.4$ .
Divide the total by the sequence length $T$ to calculate the mean: $-1.4 / 2 = -0.7$ .
Negate the mean to obtain the loss value, as the objective is to minimize the negative log-likelihood: $-(-0.7) = 0.7$ .
The final output is 0.7

Constraints:

len(log_probs) == len(returns), length >= 1.
Return the negative mean of the elementwise products.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium Entropy of a PolicyEasy

1. Background Knowledge

REINFORCE is a classic policy gradient algorithm for reinforcement learning. It directly optimizes a parameterized policy $\pi_\theta(a \mid s)$ by following the gradient of the expected return. The core insight is that the gradient of the expected return can be written as an expectation over trajectories:

$\nabla_\theta J(\theta) = \mathbb{E}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \, G_t\right]$

where $G_t$ is the return from time step $t$ (the discounted sum of future rewards). Because we want to maximize the expected return, the corresponding loss to minimize is the negative of this quantity. In practice, we use a single sampled trajectory as a Monte Carlo estimate of the expectation.

The log-probability $\log \pi(a_t \mid s_t)$ is preferred over the raw probability $\pi(a_t \mid s_t)$ for numerical stability. Taking the log converts products of small probabilities into sums, avoiding underflow in floating-point arithmetic. This is standard practice in any probabilistic model, from language models to RL policies.

The return $G_t$ acts as a weight that scales how much each action's log-probability contributes to the loss. If $G_t > 0$ , the loss encourages the policy to increase the probability of that action. If $G_t < 0$ , the loss encourages decreasing it. This is the fundamental mechanism by which the policy learns which actions lead to good outcomes.

2. Algorithm Approach

This is a straightforward element-wise multiplication followed by reduction problem. The computation follows the pattern:

Element-wise product: Multiply each log-probability by its corresponding return.
Summation: Sum all the products.
Averaging: Divide by the number of timesteps $T$ .
Negation: Apply the negative sign to convert from reward to loss.

In vectorized frameworks like PyTorch or NumPy, this can be expressed as a single expression: the negative mean of the element-wise product of the two input arrays. No loops, no conditionals, no iterative state — just a clean functional composition of basic operations.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.