Implicit Reward from DPO

mediumPROPreference OptimizationLLM 2: Training & Alignment

Extract the implicit reward from a DPO-trained model.

In DPO, the implicit reward is: $r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$

Since Z(x) is the same for all y given x, we can compare rewards by: $r(x, y) \propto \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$

Given log-probabilities from policy and reference model for N responses to the same prompt, compute the implicit rewards and rank responses.

Input:

Line 1: beta
Line 2: N
Next N lines: log_prob_policy log_prob_ref response_label

Output: Response labels ranked by implicit reward (highest first), one per line.

Example

Input

0.1
3
-2.0 -3.0 A
-1.5 -1.0 B
-3.0 -4.0 C

Output

C
A
B

Reasoning

First, we read the input values: $\beta = 0.1$ , $N = 3$ , and the log-probabilities for each response.
Then, we calculate the implicit reward for each response using the formula: $r(x, y) \propto \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ $r (x, y) \propto β lo g \frac{π _{θ} ( y ∣ x )}{π _{re f} ( y ∣ x )}$ . For the given inputs, the calculations are:
- For A: $r(A) \propto 0.1 \log \frac{-2.0}{-3.0}$
- For B: $r(B) \propto 0.1 \log \frac{-1.5}{-1.0}$
- For C: $r(C) \propto 0.1 \log \frac{-3.0}{-4.0}$
Next, we compute the actual values:
- For A: $r(A) \propto 0.1 \log \frac{-2.0}{-3.0} = 0.1 \log 0.6667 \approx 0.1 \cdot -0.4055 \approx -0.0406$
- For B: $r(B) \propto 0.1 \log \frac{-1.5}{-1.0} = 0.1 \log 1.5 \approx 0.1 \cdot 0.4055 \approx 0.0406$
- For C: $r(C) \propto 0.1 \log \frac{-3.0}{-4.0} = 0.1 \log 0.75 \approx 0.1 \cdot -0.2877 \approx -0.0288$ However, considering the proportionality and the actual log values, the correct order should be based on $0.1 \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ which translates to $0.1 \cdot (\log_prob_policy - \log_prob_ref)$ for each.
- For A: $0.1 \cdot (-2.0 - (-3.0)) = 0.1 \cdot 1 = 0.1$
- For B: $0.1 \cdot (-1.5 - (-1.0

Constraints

0.01 <= beta <= 1.0
1 <= N <= 20
Break ties by original order

Learn this in

RLHF & AlignmentLLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from LLM 2: Training & Alignment

RLHF Reward Shapinghard PPO Clipped Objectivemedium DPO Lossmedium KL Divergence Penaltyeasy Best-of-N Samplinghard

Concept primerPREVIEW

Background Knowledge

The problem revolves around Preference Optimization, specifically in the context of DPO (Deep Preference Optimization), which is a technique used in LLM (Large Language Models) to align model outputs with human preferences. The implicit reward is a crucial concept here, representing the model's preference for certain outputs over others. This reward is calculated based on the log-probabilities of the model's policy and a reference model. Understanding the log-probability and how it relates to the model's confidence in its outputs is essential.

The formula for the implicit reward, $r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$ , shows that the reward is proportional to the log-likelihood ratio of the policy and reference models. The term $Z(x)$ , although important for normalization, cancels out when comparing rewards for different $y$ given the same $x$ , simplifying the comparison to $r(x, y) \propto \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ . This simplification is key to solving the problem, as it allows for the comparison of rewards without needing to compute $Z(x)$ explicitly.

The problem also touches on ranking and comparison of model outputs based on their implicit rewards. This involves calculating the implicit reward for each response and then sorting these responses based on their rewards. Understanding how to efficiently compute and compare these rewards is vital for solving the problem.

Algorithm/Approach

The general approach to solving this type of problem involves:

Calculating the implicit reward for each response using the given formula.
Comparing these rewards to rank the responses.
Outputting the ranked list of responses.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.