Perplexity Calculator

easyFREEEvaluation MetricsLLM 3: Applications & Evaluation

Compute the perplexity of a language model on a sequence.

Perplexity is defined as: $\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i})\right)$

Given the log-probabilities of each token in a sequence, compute the perplexity.

Input: A list of log-probabilities (natural log), one per token

Output: Perplexity as a float (checked to 4 decimal places).

Example

Input

-1.0 -2.0 -1.5

Output

4.4817

Reasoning

Input: log_probs holds one natural-log probability per token, here $\log P(w_1)=-1.0$ , $\log P(w_2)=-2.0$ , $\log P(w_3)=-1.5$ , so $N=3$ . Undoing the logs, the model gave the tokens probabilities of about $0.3679$ , $0.1353$ and $0.2231$ .
Sum and average: $\sum_i \log P(w_i) = -1.0 - 2.0 - 1.5 = -4.5$ , and dividing by $N=3$ gives an average log-probability of $-1.5$ .
Negate: the minus sign in the formula turns this into an average negative log-likelihood of $+1.5$ (always $\ge 0$ , since every log-probability is $\le 0$ ).
Exponentiate: $\text{PPL} = e^{1.5} \approx 4.48169$ , which is 4.4817 to 4 decimal places.
Notice: $e^{-1.5} \approx 0.2231$ is the geometric average of the three token probabilities, and perplexity is its reciprocal: the model was, on average, about as unsure as if it were choosing uniformly among $4.48$ tokens. A perfect model (all log-probabilities $0$ ) scores exactly $1$ .

Constraints

Log-probabilities are negative (or zero)
Use natural log
Round to 4 decimal places

Learn this in

Evaluation & BenchmarksLLM Study Plan PretrainingLLM Study Plan

More from LLM 3: Applications & Evaluation

Beam Search Decoderhard INT8 Quantization Simulatormedium Nucleus (Top-P) Samplingmedium Top-K Samplingeasy Temperature Samplingeasy

Concept primer

Background Knowledge

The concept of perplexity is a fundamental evaluation metric in natural language processing (NLP), used to measure the performance of a language model. It is defined as the exponential of the average log-probability of a sequence of tokens, given by the formula: $\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i | w_{<i})\right)$ . In essence, perplexity represents how well a language model can predict a sequence of words. A lower perplexity score indicates better performance.

In the context of language models, log-probabilities play a crucial role. The log-probability of a token $w_i$ given the context $w_{<i}$ represents the probability of the model predicting that token. By summing up the log-probabilities of all tokens in a sequence and dividing by the total number of tokens $N$ , we obtain the average log-probability. The exponential of the negative average log-probability yields the perplexity.

Understanding the properties of exponential and logarithmic functions is essential for working with perplexity. The exponential function $\exp(x)$ is the inverse of the natural logarithm $\log(x)$ . This means that $\exp(\log(x)) = x$ and $\log(\exp(x)) = x$ . These properties will be useful when implementing the perplexity calculation.

Algorithm/Approach

The general approach to solving this problem involves calculating the average log-probability of a sequence of tokens and then exponentiating the negative average log-probability to obtain the perplexity. This can be achieved by iterating over the input log-probabilities, summing them up, and then applying the exponential function.

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the input log-probabilities from the space-separated input.
Initialize a variable to store the sum of log-probabilities.
Iterate over each log-probability in the input, adding it to the sum.
Calculate the average log-probability by dividing the sum by the total number of log-probabilities.
Calculate the perplexity by exponentiating the negative average log-probability.
Round the perplexity to 4 decimal places.

Common Pitfalls

When implementing the solution, watch out for the following:

Ensure that the input log-probabilities are correctly parsed and stored.
Be mindful of the order of operations when calculating the average log-probability and perplexity.
Use the correct exponential function to calculate the perplexity.

Time & Space Complexity

The time complexity of the solution is expected to be O(N), where N is the number of log-probabilities in the input. This is because we need to iterate over each log-probability once to calculate the sum. The space complexity is expected to be O(1), as we only need a constant amount of space to store the sum and the average log-probability.

solution.py

Python 3.13

Test results