📘

PPO Clipped Objective (Single Sample)

MediumReinforcement Learning 📚 RL 1: Foundations

Problem Statement

The PPO clipped surrogate for one sample, given the probability ratio r = pi_new/pi_old, advantage A, and clip range eps, is:

$L = \min\big(r\,A,\; \mathrm{clip}(r, 1-\epsilon, 1+\epsilon)\,A\big)$

Implement ppo_clip(r, adv, eps) returning the surrogate value (float). Note the min is taken after multiplying by the advantage.

Example:

Input:

ppo_clip(1.5, 2.0, 0.2)

Output:

2.4

Reasoning:

Determine the clipping bounds for the probability ratio $r$ using the range $\epsilon$ : the lower bound is $1 - 0.2 = 0.8$ and the upper bound is $1 + 0.2 = 1.2$ .
Clip the input ratio $r = 1.5$ to the calculated bounds; since $1.5 > 1.2$ , the clipped value becomes $1.2$ .
Calculate the unclipped surrogate term by multiplying the original ratio by the advantage: $1.5 \times 2.0 = 3.0$ .
Calculate the clipped surrogate term by multiplying the clipped ratio by the advantage: $1.2 \times 2.0 = 2.4$ .
Select the minimum of the two surrogate terms to ensure the objective is conservative: $\min(3.0, 2.4) = 2.4$ .
The final output is 2.4

Constraints:

r > 0, eps in (0, 1).
clip(r, 1-eps, 1+eps) bounds the ratio before the second product.
Return a float.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium Entropy of a PolicyEasy

1. Background Knowledge

Proximal Policy Optimization (PPO) is a widely used policy gradient algorithm that improves upon vanilla policy gradient methods by constraining policy updates to prevent large, destabilizing changes. The core idea is to maximize a clipped surrogate objective that penalizes the new policy $\pi_{\text{new}}$ from deviating too far from the old policy $\pi_{\text{old}}$ . The deviation is measured by the probability ratio $r = \frac{\pi_{\text{new}}(a|s)}{\pi_{\text{old}}(a|s)}$ , which quantifies how much more (or less) likely the new policy makes an action compared to the old one.

The advantage function $A(s,a)$ estimates how much better an action is compared to the average action in a given state. It is typically computed using methods like Generalized Advantage Estimation (GAE). A positive advantage indicates the action is better than average, while a negative advantage indicates it is worse. The surrogate objective multiplies the ratio by the advantage to scale the update direction and magnitude.

The clipping mechanism is the hallmark of PPO. Instead of allowing $r$ to grow unbounded, it is clipped to the interval $[1-\epsilon, 1+\epsilon]$ . The final objective takes the minimum of the unclipped term $rA$ and the clipped term $\text{clip}(r, 1-\epsilon, 1+\epsilon)A$ . This ensures that if the new policy is too far from the old one in a direction that would increase the objective, the gradient is zeroed out, effectively preventing overly aggressive updates.

2. Algorithm Approach

The problem requires implementing a single-sample evaluation of the PPO clipped surrogate. The approach is straightforward:

Compute the clipped ratio by constraining $r$ to the range $[1-\epsilon, 1+\epsilon]$ .
Calculate two candidate values:

The unclipped surrogate: $r \times A$
The clipped surrogate: $\text{clip}(r) \times A$

Return the minimum of these two values.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.