Return Standardization Baseline
Problem Statement
A common variance-reduction trick standardizes the batch of returns before using them as weights:
G^i​=σ+ϵGi​−μ​
where mu and sigma are the mean and (population) standard deviation of the returns, and epsilon guards against divide-by-zero. Implement standardize_returns(returns, eps) returning the standardized list.
Example:
standardize_returns([1.0, 2.0, 3.0], 1e-8)
[-1.2247, 0.0, 1.2247]
- Calculate the mean (μ) of the returns [1.0,2.0,3.0] to determine the center of the distribution: μ=31.0+2.0+3.0​=2.0.
- Compute the population variance by averaging the squared deviations from the mean: var=3(1.0−2.0)2+(2.0−2.0)2+(3.0−2.0)2​=31+0+1​=32​≈0.6667.
- Determine the standard deviation (σ) by taking the square root of the variance: σ=32​​≈0.8165.
- Standardize each return value by subtracting the mean and dividing by the sum of the standard deviation and the epsilon guard (10−8), which is effectively just σ:
- For 1.0: 0.8165+10−81.0−2.0​≈−1.2247
- For 2.0: 0.8165+10−82.0−2.0​=0.0
- For 3.0: 0.8165+10−83.0−2.0​≈1.2247
- The final output is [-1.2247, 0.0, 1.2247]
Constraints:
len(returns) >= 1.- Use the population standard deviation (divide by N).
- Return a list of floats.
1. Background Knowledge
In policy gradient methods, the update direction is often scaled by the return (or advantage) of each sampled trajectory. Raw returns can have high variance and arbitrary scale, which makes gradient steps unstable. A widely used variance-reduction trick is to standardize the batch of returns: subtract the batch mean and divide by the batch standard deviation. This maps the batch to approximately zero mean and unit variance, making the learning rate more interpretable and the optimization landscape smoother.
The standardization formula is:
G^i​=σ+ϵGi​−μ​where μ is the population mean of the batch and σ is the population standard deviation. The small constant ϵ (often 10−8) prevents division by zero when all returns are identical. Note that this is a batch-level normalization, not a per-sample operation; it relies on statistics computed over the entire input list.
In practice, standardized returns are frequently used as weights in REINFORCE and A2C-style algorithms. While more sophisticated baselines (e.g., learned value functions) exist, simple standardization is a cheap, model-free way to reduce variance without extra parameters.
2. Algorithm Approach
This is a straightforward descriptive statistics computation followed by an element-wise transformation. The pattern is:
- Compute the mean of the input list.
- Compute the population standard deviation (divide by n, not n−1).
- Apply the standardization formula to each element.
No sorting, recursion, or data structures beyond a single pass (or two passes) are needed. The key is to use the correct definition of standard deviation and handle the edge case where σ=0.
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.