Reward Model Loss

easyFREEReward ModelingLLM 2: Training & Alignment

Compute the reward model training loss.

Given N preference pairs (chosen reward, rejected reward), the loss is: $L = -\frac{1}{N} \sum_{i=1}^{N} \log \sigma(r_{\text{chosen}_i} - r_{\text{rejected}_i})$

where σ is the sigmoid function.

Input:

Line 1: N (number of pairs)
Next N lines: r_chosen r_rejected

Output: The loss value, rounded to 4 decimal places.

Example

Input

2
2.0 1.0
1.5 0.5

Output

0.2899

Reasoning

We start with the given input: $N = 2$ , and two pairs of rewards: $(2.0, 1.0)$ and $(1.5, 0.5)$ .
For each pair, we calculate $\log \sigma(r_{\text{chosen}_i} - r_{\text{rejected}_i})$ $lo g σ (r_{chosen_{i}} - r_{rejected_{i}})$ :
- For the first pair: $\log \sigma(2.0 - 1.0) = \log \sigma(1.0)$
- For the second pair: $\log \sigma(1.5 - 0.5) = \log \sigma(1.0)$
We then compute the average of these values and multiply by $-1$ : $-\frac{1}{2} (\log \sigma(1.0) + \log \sigma(1.0))$
Since $\sigma(1.0) \approx 0.7311$ , $\log \sigma(1.0) \approx -0.2899 \times 2$ , and thus $-\frac{1}{2} \times -0.2899 \times 2 = 0.2899$

Constraints

1 <= N <= 100
Use log-sigmoid: log(σ(x)) = x - log(1 + exp(x)) for stability when x > 0
Round to 4 decimal places

Learn this in

RLHF & AlignmentLLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from LLM 2: Training & Alignment

RLHF Reward Shapinghard Implicit Reward from DPOmedium PPO Clipped Objectivemedium DPO Lossmedium KL Divergence Penaltyeasy

Concept primer

Background Knowledge

The problem is related to reward modeling, which is a crucial aspect of training large language models (LLMs). In this context, a reward model is used to predict the quality or usefulness of a particular response or action. The goal is to train the model to prefer certain responses over others based on human preferences. The sigmoid function, denoted by $\sigma$ , is a mathematical function that maps any real-valued number to a value between 0 and 1. It is often used in machine learning models, particularly in binary classification problems, where the goal is to predict one of two classes.

The reward model training loss is a measure of how well the model is doing in terms of predicting the preferred responses. The loss function given in the problem description is a specific type of loss function, which is based on the logistic function. This loss function is designed to encourage the model to produce higher scores for the chosen rewards and lower scores for the rejected rewards. The log function is used to penalize the model for producing scores that are not consistent with the human preferences.

In the context of machine learning, the goal is to minimize the loss function with respect to the model's parameters. This is typically done using optimization algorithms, such as gradient descent, which iteratively update the model's parameters to reduce the loss. Understanding the mathematical properties of the loss function, including its differentiability and convexity, is essential for designing effective optimization algorithms.

Algorithm/Approach

The general approach to solving this problem involves computing the loss function for a given set of preference pairs. This can be done by iterating over each pair, computing the difference between the chosen and rejected rewards, applying the sigmoid function, and then computing the log of the result. The average loss over all pairs can then be computed by summing up the individual losses and dividing by the total number of pairs.

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the input data, including the number of pairs and the chosen and rejected rewards for each pair.
Initialize a variable to store the total loss.
Iterate over each pair, and for each pair:
- Compute the difference between the chosen and rejected rewards.
- Apply the sigmoid function to the difference.
- Compute the log of the result.
- Add the log value to the total loss.
Divide the total loss by the number of pairs to get the average loss.
Round the average loss to 4 decimal places.

Common Pitfalls

When implementing the solution, watch out for the following common pitfalls:

Make sure to use the correct implementation of the sigmoid function, which is $\sigma(x) = \frac{1}{1 + e^{-x}}$ .
Be careful when computing the log of the sigmoid function, as this can result in very small values that may cause numerical instability.
Make sure to initialize the total loss variable to zero before iterating over the pairs.

Time & Space Complexity

The time complexity of the solution is O(N), where N is the number of pairs, since we need to iterate over each pair to compute the loss. The space complexity is O(1), since we only need to store a few variables to compute the loss, regardless of the size of the input.

solution.py

Python 3.13

Test results