Return Standardization Baseline

mediumPROPolicy Gradients and Advantage EstimationReinforcement LearningRL 1: Foundations

Problem Statement

A common variance-reduction trick standardizes the batch of returns before using them as weights:

$\hat{G}_i = \frac{G_i - \mu}{\sigma + \epsilon}$

where mu and sigma are the mean and (population) standard deviation of the returns, and epsilon guards against divide-by-zero. Implement standardize_returns(returns, eps) returning the standardized list.

Example

Input

standardize_returns([1.0, 2.0, 3.0], 1e-8)

Output

[-1.2247, 0.0, 1.2247]

Reasoning

Calculate the mean ( $\mu$ ) of the returns $[1.0, 2.0, 3.0]$ to determine the center of the distribution: $\mu = \frac{1.0 + 2.0 + 3.0}{3} = 2.0$ .
Compute the population variance by averaging the squared deviations from the mean: $\text{var} = \frac{(1.0 - 2.0)^2 + (2.0 - 2.0)^2 + (3.0 - 2.0)^2}{3} = \frac{1 + 0 + 1}{3} = \frac{2}{3} \approx 0.6667$ .
Determine the standard deviation ( $\sigma$ ) by taking the square root of the variance: $\sigma = \sqrt{\frac{2}{3}} \approx 0.8165$ .
Standardize each return value by subtracting the mean and dividing by the sum of the standard deviation and the epsilon guard ( $10^{-8}$ $1 0^{- 8}$ ), which is effectively just $\sigma$ $σ$ :
- For $1.0$ : $\frac{1.0 - 2.0}{0.8165 + 10^{-8}} \approx -1.2247$
- For $2.0$ : $\frac{2.0 - 2.0}{0.8165 + 10^{-8}} = 0.0$
- For $3.0$ : $\frac{3.0 - 2.0}{0.8165 + 10^{-8}} \approx 1.2247$
The final output is [-1.2247, 0.0, 1.2247]

Constraints

len(returns) >= 1.
Use the population standard deviation (divide by N).
Return a list of floats.

Learn this in

Policy Gradient MethodsReinforcement Learning Actor-Critic & PPOReinforcement Learning

More from RL 1: Foundations

GAE Over a Full Trajectoryhard PPO Clipped Objective (Single Sample)medium TD Residuals for GAEmedium REINFORCE Loss Termmedium Entropy of a Policyeasy

Concept primerPREVIEW

1. Background Knowledge

In policy gradient methods, the update direction is often scaled by the return (or advantage) of each sampled trajectory. Raw returns can have high variance and arbitrary scale, which makes gradient steps unstable. A widely used variance-reduction trick is to standardize the batch of returns: subtract the batch mean and divide by the batch standard deviation. This maps the batch to approximately zero mean and unit variance, making the learning rate more interpretable and the optimization landscape smoother.

The standardization formula is:

\hat{G}_i = \frac{G_i - \mu}{\sigma + \epsilon}

where $\mu$ is the population mean of the batch and $\sigma$ is the population standard deviation. The small constant $\epsilon$ (often $10^{-8}$ ) prevents division by zero when all returns are identical. Note that this is a batch-level normalization, not a per-sample operation; it relies on statistics computed over the entire input list.

In practice, standardized returns are frequently used as weights in REINFORCE and A2C-style algorithms. While more sophisticated baselines (e.g., learned value functions) exist, simple standardization is a cheap, model-free way to reduce variance without extra parameters.

2. Algorithm Approach

This is a straightforward descriptive statistics computation followed by an element-wise transformation. The pattern is:

Compute the mean of the input list.
Compute the population standard deviation (divide by $n$ , not $n-1$ ).
Apply the standardization formula to each element.

No sorting, recursion, or data structures beyond a single pass (or two passes) are needed. The key is to use the correct definition of standard deviation and handle the edge case where $\sigma = 0$ .

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results