PPO Clipped Objective
Compute the PPO clipped surrogate objective.
The PPO clip objective for a single sample: LCLIP=min(rt⋅At,clip(rt,1−ϵ,1+ϵ)⋅At)
where r_t = π_θ(a|s) / π_old(a|s) is the probability ratio, A_t is the advantage, and ε is the clip range.
Input:
- Line 1: epsilon (clip range)
- Line 2: N (number of samples)
- Next N lines: ratio advantage
Output: Average clipped objective, rounded to 4 decimal places.
Example:
0.2 3 1.5 1.0 0.8 -0.5 1.0 2.0
1.1833
- The clip range ϵ is set to 0.2, which means the clipped ratio will be between 1−ϵ=0.8 and 1+ϵ=1.2.
- For each sample, we calculate the clipped surrogate objective LCLIP:
- For the first sample, rt=1.5, At=1.0, so LCLIP=min(1.5⋅1.0,clip(1.5,0.8,1.2)⋅1.0)=min(1.5,1.2⋅1.0)=1.2.
- For the second sample, rt=0.8, At=−0.5, so LCLIP=min(0.8⋅−0.5,clip(0.8,0.8,1.2)⋅−0.5)=min(−0.4,0.8⋅−0.5)=−0.4.
- For the third sample, rt=1.0, At=2.0, so LCLIP=min(1.0⋅2.0,clip(1.0,0.8,1.2)⋅2.0)=min(2.0,1.0⋅2.0)=2.0.
- We then calculate the average clipped objective: (1.2+(−0.4)+2.0)/3=1.0+0.1833=1.1833.
- The final output is 1.1833, rounded to 4 decimal places.
Constraints:
- 0.1 <= epsilon <= 0.3
- 1 <= N <= 50
- The objective should be MAXIMIZED (we output positive for good updates)
- Round to 4 decimal places
More from LLM 2: Training & Alignment
Background Knowledge
The PPO Clipped Objective is a key component in the Proximal Policy Optimization (PPO) algorithm, which is a model-free, on-policy reinforcement learning method. PPO is designed to improve the stability and performance of policy optimization by using a trust region method, where the new policy is constrained to be close to the old policy. The probability ratio, rt=πold(a∣s)πθ(a∣s), represents the ratio of the new policy to the old policy for a given action a in state s. The advantage, At, represents the benefit of taking action a in state s under the current policy.
The clip range, ϵ, is a hyperparameter that controls the trust region by clipping the probability ratio. This prevents large updates to the policy, which can lead to instability. The PPO clipped surrogate objective, LCLIP, is a lower bound on the true objective function, which makes it a suitable surrogate objective for optimization. By minimizing LCLIP, we can improve the policy while ensuring that it remains close to the old policy.
In the context of preference optimization, the PPO clipped objective is used to optimize the policy based on preferences or rewards. The goal is to learn a policy that maximizes the cumulative reward over time. The PPO clipped objective provides a stable and efficient way to optimize the policy, even in complex environments with high-dimensional state and action spaces.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.