📘

Perplexity Calculator

EasyEvaluation Metrics 📚 LLM 3: Applications & Evaluation

Compute the perplexity of a language model on a sequence.

Perplexity is defined as: $\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i})\right)$

Given the log-probabilities of each token in a sequence, compute the perplexity.

Input: Space-separated log-probabilities (natural log)

Output: Perplexity, rounded to 4 decimal places.

Example:

Input:

-1.0 -2.0 -1.5

Output:

4.4817

Reasoning:

First, we calculate the sum of the log-probabilities: $-1.0 + (-2.0) + (-1.5) = -4.5$
Then, we calculate the average log-probability: $-\frac{1}{3} \cdot -4.5 = 1.5$
Next, we apply the exponential function to get the perplexity: $\exp(-1.5) = 0.2231$ is incorrect, we actually need to use $-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i}) = -\frac{1}{3} \cdot (-4.5) = 1.5$ , then $\exp(-1.5)$ is not the correct step. The correct step is to calculate $\exp(-(-1.5))$ since the formula is $\exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i})\right)$ and $-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i}) = 1.5$ . So, $\exp(1.5)$
The final output is $\exp(1.5) \approx 4.4817$ , rounded to 4 decimal places.

Constraints:

Log-probabilities are negative (or zero)
Use natural log
Round to 4 decimal places

More from LLM 3: Applications & Evaluation

Beam Search DecoderHard INT8 Quantization SimulatorMedium Nucleus (Top-P) SamplingMedium Top-K SamplingEasy Temperature SamplingEasy

Background Knowledge

The concept of perplexity is a fundamental evaluation metric in natural language processing (NLP), used to measure the performance of a language model. It is defined as the exponential of the average log-probability of a sequence of tokens, given by the formula: $\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i})\right)$ . In essence, perplexity represents how well a language model can predict a sequence of words. A lower perplexity score indicates better performance.

In the context of language models, log-probabilities play a crucial role. The log-probability of a token $w_i$ given the context $w_{<i}$ represents the probability of the model predicting that token. By summing up the log-probabilities of all tokens in a sequence and dividing by the total number of tokens $N$ , we obtain the average log-probability. The exponential of the negative average log-probability yields the perplexity.

Understanding the properties of exponential and logarithmic functions is essential for working with perplexity. The exponential function $\exp(x)$ is the inverse of the natural logarithm $\log(x)$ . This means that $\exp(\log(x)) = x$ and $\log(\exp(x)) = x$ . These properties will be useful when implementing the perplexity calculation.

Algorithm/Approach

The general approach to solving this problem involves calculating the average log-probability of a sequence of tokens and then exponentiating the negative average log-probability to obtain the perplexity. This can be achieved by iterating over the input log-probabilities, summing them up, and then applying the exponential function.

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the input log-probabilities from the space-separated input.
Initialize a variable to store the sum of log-probabilities.
Iterate over each log-probability in the input, adding it to the sum.
Calculate the average log-probability by dividing the sum by the total number of log-probabilities.
Calculate the perplexity by exponentiating the negative average log-probability.
Round the perplexity to 4 decimal places.

Common Pitfalls

When implementing the solution, watch out for the following:

Ensure that the input log-probabilities are correctly parsed and stored.
Be mindful of the order of operations when calculating the average log-probability and perplexity.
Use the correct exponential function to calculate the perplexity.

Time & Space Complexity

The time complexity of the solution is expected to be O(N), where N is the number of log-probabilities in the input. This is because we need to iterate over each log-probability once to calculate the sum. The space complexity is expected to be O(1), as we only need a constant amount of space to store the sum and the average log-probability.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.