DPO Loss

mediumPROPreference OptimizationLLM 2: Training & Alignment

Compute the Direct Preference Optimization (DPO) loss.

DPO loss for a single preference pair: $L_{DPO} = -\log \sigma(\beta \cdot (\log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}))$

where y_w is the preferred (winning) response and y_l is the rejected (losing) response.

Input:

Line 1: beta (temperature parameter)
Line 2: N (number of preference pairs)
Next N lines: log_ratio_w log_ratio_l (log probability ratios for chosen and rejected)

where log_ratio = log(π_θ(y|x) / π_ref(y|x))

Output: Average DPO loss, rounded to 4 decimal places.

Example

Input

0.1
2
1.0 -1.0
0.5 0.0

Output

0.4887

Reasoning

The temperature parameter $\beta$ is set to $0.1$ and the number of preference pairs $N$ is set to $2$ .
For each preference pair, we calculate the DPO loss using the given formula:
- For the first pair, $L_{DPO} = -\log \sigma(0.1 \cdot (1.0 - (-1.0))) = -\log \sigma(0.2)$
- For the second pair, $L_{DPO} = -\log \sigma(0.1 \cdot (0.5 - 0.0)) = -\log \sigma(0.05)$
We compute the values: $-\log \sigma(0.2) \approx 0.5988$ and $-\log \sigma(0.05) \approx 0.3785$
The average DPO loss is then calculated as $(0.5988 + 0.3785) / 2 \approx 0.4887$ , which is rounded to $4$ decimal places to obtain the final output.

Constraints

0.01 <= beta <= 1.0
1 <= N <= 50
Round to 4 decimal places

Learn this in

RLHF & AlignmentLLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from LLM 2: Training & Alignment

RLHF Reward Shapinghard Implicit Reward from DPOmedium PPO Clipped Objectivemedium KL Divergence Penaltyeasy Best-of-N Samplinghard

Concept primerPREVIEW

Background Knowledge

The problem involves computing the Direct Preference Optimization (DPO) loss, a concept from the field of preference optimization. In this context, the goal is to optimize a model based on user preferences, which are provided as pairs of preferred and rejected responses. The DPO loss is a measure of how well the model aligns with these preferences. The formula for DPO loss involves the logistic function ( $\sigma$ ), which is commonly used in machine learning to model binary outcomes. The temperature parameter ( $\beta$ ) controls the sensitivity of the logistic function.

The DPO loss formula also involves log probability ratios, which represent the difference in log probabilities between the model's predictions and a reference model's predictions. These log ratios are used to compute the log odds of the preferred response versus the rejected response. The logistic function is then applied to these log odds to obtain a probability value between 0 and 1. The DPO loss is the negative log of this probability value, which encourages the model to produce high probabilities for the preferred responses.

To understand this problem, it's essential to have a basic knowledge of probability theory, logistic functions, and machine learning concepts, such as loss functions and optimization. Familiarity with Python programming is also necessary to implement the solution.

Algorithm/Approach

The general approach to solving this problem involves:

Reading the input parameters, including the temperature parameter ( $\beta$ ) and the number of preference pairs ( $N$ )
Computing the DPO loss for each preference pair using the provided formula
Averaging the DPO losses over all preference pairs to obtain the final result

This approach can be implemented using a simple iterative algorithm, where each iteration computes the DPO loss for a single preference pair.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results