PPO Clipped Surrogate Objective

Problem Statement

Compute PPO's clipped surrogate objective over a minibatch, along with the fraction of samples the clip range actually touched.

Background

Vanilla policy gradients are only valid for data collected by the current policy, so a step that is too large destroys the policy and there is no gradient signal left to recover with. TRPO solved this with a hard KL trust region and a conjugate-gradient solve. PPO gets most of the benefit with a first-order objective you can write in four lines.

Start from the importance-sampling ratio between the new and old policy on the same action:

$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{old}}(a_t \mid s_t)} = \exp\big(\log \pi_\theta - \log \pi_{\theta_{old}}\big)$

Computing it as exp of a log-difference rather than a quotient of probabilities is what keeps it numerically sane. The objective is then

$L^{CLIP} = \mathbb{E}_t\Big[\min\big(r_t A_t,\ \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon)\, A_t\big)\Big]$

The min makes this a pessimistic lower bound, and its asymmetry is the whole trick:

• When $A_t > 0$ the objective stops rewarding you once $r_t > 1+\varepsilon$ — no more gain from cranking a good action's probability up.
• When $A_t < 0$ the clip binds on the other side, at $1-\varepsilon$.
• Crucially, when $A_t < 0$ and $r_t$ is large, the min picks the unclipped term (it is more negative). The objective deliberately leaves the gradient un-truncated so the policy can pull itself back — clipping there would strand it.

The commonly logged diagnostic clipfrac is the fraction of samples with $|r_t - 1| > \varepsilon$; a value that climbs toward 1 means the policy is moving too far per update.

Your Task

Implement:

def ppo_clip_objective(logp_old, logp_new, advantages, clip_eps):
    ...

• logp_old[i], logp_new[i] — log-probabilities of the same action under the old and new policy.
• advantages[i] — the advantage estimate.
• clip_eps — the clip range, e.g. 0.2.

Return a **tuple **(objective, clip_fraction)****:

• objective — the mean over the batch of min(ratio*A, clip(ratio, 1-eps, 1+eps)*A).
• clip_fraction — the fraction of samples with abs(ratio - 1) > clip_eps.

Both are floats.

Input / Output Format

Three lists of floats and a float in; a tuple of two floats out, rounded to 4 decimals by the grader.

Sample

obj, frac = ppo_clip_objective([0.0, 0.0], [0.0, 0.0], [2.0, -3.0], 0.2)
print(round(obj, 4), round(frac, 4))

Output:

-0.5 0.0

Both ratios are exp(0) = 1, nothing is clipped, and the objective is the mean advantage (2.0 - 3.0)/2 = -0.5.