📘

PPO Clipped Surrogate Objective

HardReinforcement Learning 📚 RL 1: Foundations

Problem Statement

Compute PPO's clipped surrogate objective over a minibatch, along with the fraction of samples the clip range actually touched.

Background

Vanilla policy gradients are only valid for data collected by the current policy, so a step that is too large destroys the policy and there is no gradient signal left to recover with. TRPO solved this with a hard KL trust region and a conjugate-gradient solve. PPO gets most of the benefit with a first-order objective you can write in four lines.

Start from the importance-sampling ratio between the new and old policy on the same action:

$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)} = \exp\big(\log \pi_\theta - \log \pi_{\theta_{old}}\big)$

Computing it as exp of a log-difference rather than a quotient of probabilities is what keeps it numerically sane. The objective is then

$L^{CLIP} = \mathbb{E}_t\Big[\min\big(r_t A_t,\ \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon)\, A_t\big)\Big]$

The min makes this a pessimistic lower bound, and its asymmetry is the whole trick:

When $A_t > 0$ the objective stops rewarding you once $r_t > 1+\varepsilon$ — no more gain from cranking a good action's probability up.
When $A_t < 0$ the clip binds on the other side, at $1-\varepsilon$ .
Crucially, when $A_t < 0$ and $r_t$ is large, the min picks the unclipped term (it is more negative). The objective deliberately leaves the gradient un-truncated so the policy can pull itself back — clipping there would strand it.

The commonly logged diagnostic clipfrac is the fraction of samples with $|r_t - 1| > \varepsilon$ ; a value that climbs toward 1 means the policy is moving too far per update.

Your Task

Implement:

def ppo_clip_objective(logp_old, logp_new, advantages, clip_eps):
    ...

logp_old[i], logp_new[i] — log-probabilities of the same action under the old and new policy.
advantages[i] — the advantage estimate.
clip_eps — the clip range, e.g. 0.2.

Return a **tuple **(objective, clip_fraction)****:

objective — the mean over the batch of min(ratio*A, clip(ratio, 1-eps, 1+eps)*A).
clip_fraction — the fraction of samples with abs(ratio - 1) > clip_eps.

Both are floats.

Input / Output Format

Three lists of floats and a float in; a tuple of two floats out, rounded to 4 decimals by the grader.

Sample

obj, frac = ppo_clip_objective([0.0, 0.0], [0.0, 0.0], [2.0, -3.0], 0.2)
print(round(obj, 4), round(frac, 4))

Output:

-0.5 0.0

Both ratios are exp(0) = 1, nothing is clipped, and the objective is the mean advantage (2.0 - 3.0)/2 = -0.5.

Example:

Input:

ppo_clip_objective([0.0, 0.0], [0.0, 0.0], [2.0, -3.0], 0.2)

Output:

-0.5 0.0

Reasoning:

Identical log-probabilities give ratio = exp(0) = 1 for both samples, so nothing falls outside [0.8, 1.2] and clip_fraction is 0. With ratio 1 the clipped and unclipped terms are both just the advantage, so the objective is the mean advantage (2.0 + (-3.0))/2 = -0.5.

Constraints:

1 <= batch_size <= 100000
0.0 < clip_eps < 1.0
Compute the ratio as exp(logp_new - logp_old), not as a quotient of probabilities.
objective is a mean, not a sum.
clip_fraction counts samples with abs(ratio - 1) > clip_eps, regardless of the sign of the advantage.
Do not round inside the function; return (objective, clip_fraction) in that order.

More from RL 1: Foundations

GAE Over a Full TrajectoryHard PPO Clipped Objective (Single Sample)Medium TD Residuals for GAEMedium Return Standardization BaselineMedium REINFORCE Loss TermMedium

1. Background Knowledge

Proximal Policy Optimization (PPO) is a popular reinforcement learning algorithm designed to update policies safely. Unlike vanilla policy gradients, which can become unstable if the new policy diverges too far from the old one, PPO uses a clipped surrogate objective to limit the magnitude of policy updates. This prevents the agent from making drastic changes that could destroy previously learned behaviors.

The core mechanism relies on the importance sampling ratio, $r_t(\theta)$ , which measures how much the probability of an action has changed between the old and new policies. By computing this ratio as the exponential of the difference in log-probabilities, we ensure numerical stability. The objective function then takes the minimum of the unclipped ratio weighted by the advantage and the clipped ratio weighted by the advantage. This creates a pessimistic lower bound that encourages the policy to improve actions with positive advantages while preventing it from degrading actions with negative advantages too aggressively.

The clip fraction is a diagnostic metric indicating how many samples were affected by the clipping mechanism. A high clip fraction suggests the policy is moving too far per update, potentially indicating that the learning rate is too high or the clip range is too small. Understanding this metric helps in tuning hyperparameters and diagnosing training stability.

2. Algorithm Approach

The problem requires implementing the core mathematical operations of PPO's clipped surrogate objective. The approach involves vectorized operations to efficiently compute the objective and clip fraction for a batch of samples.

Compute Importance Ratios: Calculate the ratio $r_t$ for each sample using the exponential of the difference between new and old log-probabilities.
Apply Clipping: Clip the ratios to the range $[1-\varepsilon, 1+\varepsilon]$ .
Compute Surrogate Terms: Calculate both the unclipped term ( $r_t \cdot A_t$ ) and the clipped term ( $\text{clip}(r_t) \cdot A_t$ ).
Select Minimum: For each sample, take the minimum of the two terms to form the final objective contribution.
Aggregate Results: Compute the mean of the selected terms for the objective and the fraction of samples where clipping occurred for the clip fraction.

This approach leverages the element-wise operations available in numerical libraries like NumPy to handle the batch processing efficiently.

3. Step-by-Step Strategy

Import NumPy: Use numpy for efficient array operations. Convert input lists to NumPy arrays if they aren't already.
Calculate Log-Probability Difference: Compute the difference between logp_new and logp_old for each sample.
Compute Ratios: Apply the exponential function to the differences to get the importance sampling ratios $r_t$ .
Determine Clip Bounds: Define the lower bound as $1 - \text{clip\_eps}$ and the upper bound as $1 + \text{clip\_eps}$ .
Clip Ratios: Use a clipping function to restrict the ratios to the defined bounds.
Compute Objective Terms:

Calculate the unclipped surrogate: $r_t \cdot \text{advantages}$ .
Calculate the clipped surrogate: $\text{clipped\_ratios} \cdot \text{advantages}$ .

Select Minimum: Use a minimum function to select the smaller value between the unclipped and clipped surrogates for each sample.
Calculate Objective: Compute the mean of the selected minimum values.
Calculate Clip Fraction: Determine the fraction of samples where the absolute difference between the original ratio and 1 exceeds clip_eps.
Return Results: Return the tuple (objective, clip_fraction).

4. Common Pitfalls

Numerical Stability: Always compute the ratio as exp(logp_new - logp_old) rather than exp(logp_new) / exp(logp_old). The latter can lead to overflow or underflow issues with extreme log-probability values.
Clipping Logic: Ensure the clipping is applied to the ratio $r_t$ before multiplying by the advantage. The clip bounds are relative to 1, not 0.
Clip Fraction Calculation: The clip fraction is based on the original ratio $r_t$ , not the clipped ratio. It measures how many samples were actually clipped, i.e., $|r_t - 1| > \varepsilon$ .
Mean vs. Sum: The objective is the mean over the batch, not the sum. Ensure you divide by the number of samples.
Data Types: Ensure all inputs are treated as floats to avoid integer division issues or type errors in numerical operations.

5. Time & Space Complexity

Time Complexity: The algorithm performs a constant number of element-wise operations on arrays of size $N$ (the batch size). Each operation (subtraction, exponential, clipping, multiplication, minimum, mean) takes $O(N)$ time. Therefore, the overall time complexity is $O(N)$ .
Space Complexity: The algorithm requires additional space to store intermediate arrays such as the ratios, clipped ratios, and surrogate terms. Each of these arrays has size $N$ . Thus, the space complexity is $O(N)$ . If in-place operations are used or intermediate arrays are reused, the space complexity can be reduced, but typically $O(N)$ is expected for clarity and safety.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.