Pairwise Ranking Accuracy

mediumPROReward ModelingLLM 2: Training & Alignment

Evaluate a reward model's pairwise ranking accuracy.

Given N preference pairs with true labels and predicted reward scores, compute:

The fraction of pairs where the model correctly ranks chosen > rejected
The average margin (r_chosen - r_rejected) for correct predictions
The average margin for incorrect predictions

Input:

Line 1: N
Next N lines: r_chosen r_rejected (predicted rewards for the chosen and rejected responses)

A prediction is correct if r_chosen > r_rejected.

Output:

Line 1: Accuracy (fraction correct), rounded to 4 decimal places
Line 2: Average margin for correct predictions, rounded to 4 decimal places (or 0.0000 if none)
Line 3: Average margin for incorrect predictions, rounded to 4 decimal places (or 0.0000 if none)

Example

Input

4
2.0 1.0
1.5 2.5
3.0 0.0
0.5 0.5

Output

0.5000
2.0000
-0.7500

Reasoning

The input contains 4 pairs of predicted rewards, which are evaluated to determine the correctness of each pair:
- Pair 1: $2.0 > 1.0$ , correct
- Pair 2: $1.5 < 2.5$ , incorrect
- Pair 3: $3.0 > 0.0$ , correct
- Pair 4: $0.5 = 0.5$ , incorrect (since $r_{chosen} \ngtr r_{rejected}$ )
The accuracy is calculated as the fraction of correct pairs: $\frac{2}{4} = 0.5$
The average margin for correct predictions is calculated: $\frac{(2.0-1.0) + (3.0-0.0)}{2} = \frac{1.0 + 3.0}{2} = \frac{4.0}{2} = 2.0$
The average margin for incorrect predictions is calculated: $\frac{(1.5-2.5) + (0.5-0.5)}{2} = \frac{-1.0 + 0.0}{2} = \frac{-1.0}{2} = -0.5$ , but since there are two incorrect pairs and one has a margin of 0, the actual calculation is $\frac{-1.0}{1} = -1.0$ for the non-zero margin, and then considering the equal rewards pair, it results in $\frac{-1.0 + 0}{2} = -0.5$ , however the provided output suggests considering only the non-zero margin pair and another incorrect pair with a margin, hence $\frac{-1.0}{1} = -1.0$ is not the correct calculation and $\frac{(1.5-2.5)}{1} = -1.0$ and the other incorrect pair has $0.5-0.5=0$ , but the output $-0.75$ implies $\frac{-1.0+(-0.5)}{2} = \frac{-1.5}{2} = -0.75$ which matches the given output when considering both incorrect pairs have non-zero margins or one of them has a margin of $-0.5$ which is not the case, so the provided calculation in this step might be incorrect and the

Constraints

1 <= N <= 100
If r_chosen == r_rejected, count as incorrect
Round to 4 decimal places

Learn this in

Evaluation & BenchmarksLLM Study Plan RLHF & AlignmentLLM Study Plan

More from LLM 2: Training & Alignment

RLHF Reward Shapinghard Implicit Reward from DPOmedium PPO Clipped Objectivemedium DPO Lossmedium KL Divergence Penaltyeasy

Concept primerPREVIEW

Background Knowledge

The problem of Pairwise Ranking Accuracy is rooted in the concept of reward modeling, which is a crucial aspect of training and aligning large language models (LLMs). Reward modeling involves designing a system that can evaluate the quality of the model's outputs and provide feedback in the form of rewards or penalties. This feedback loop is essential for improving the model's performance over time. In the context of pairwise ranking, the model is presented with pairs of responses and must predict which one is preferred or of higher quality.

The key concept here is ranking accuracy, which measures how well the model can distinguish between the preferred (chosen) and non-preferred (rejected) responses. This is typically evaluated using metrics such as the fraction of correct predictions and the margin between the predicted rewards for the chosen and rejected responses. Understanding how to calculate and interpret these metrics is essential for evaluating the performance of a reward model.

In the context of machine learning, pairwise ranking is a type of supervised learning problem, where the model is trained on labeled data (in this case, pairs of responses with true labels) to learn a ranking function. The goal is to learn a function that can accurately predict the preferred response in a pair, given the input features (e.g., the text of the responses). This requires a deep understanding of optimization algorithms, loss functions, and evaluation metrics, which are all critical components of machine learning pipelines.

Algorithm/Approach

The general approach to solving this problem involves the following algorithm pattern:

Read in the input data (number of pairs and predicted rewards for each pair)
Initialize variables to track the number of correct predictions, total predictions, and sum of margins for correct and incorrect predictions
Iterate over each pair of predicted rewards and calculate the margin (difference between the predicted rewards)
Check if the prediction is correct (i.e., the predicted reward for the chosen response is higher than the predicted reward for the rejected response)
Update the tracking variables accordingly
Calculate the accuracy, average margin for correct predictions, and average margin for incorrect predictions

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results