BLEU-4 Score

hardPROEvaluation MetricsLLM 3: Applications & Evaluation

Compute the full BLEU-4 score.

BLEU-4 uses n-gram precisions for n=1,2,3,4: $\text{BLEU-4} = BP \cdot \exp\left(\sum_{n=1}^{4} \frac{1}{4} \log p_n\right)$

where p_n is the modified n-gram precision (clipped counts / candidate n-gram count) and BP is the brevity penalty.

If any p_n = 0, BLEU-4 = 0.

Input:

Line 1: reference
Line 2: candidate

Output: BLEU-4 score, rounded to 4 decimal places.

Example

Input

the cat is on the mat
the cat sat on the mat

Output

0.6687

Reasoning

First, we calculate the modified n-gram precisions ( $p_n$ $p_{n}$ ) for $n=1,2,3,4$ $n = 1, 2, 3, 4$ by comparing the reference and candidate sentences:
- $p_1 = \frac{\text{clipped unigram count}}{\text{candidate unigram count}} = \frac{7}{7} = 1$
- $p_2 = \frac{\text{clipped bigram count}}{\text{candidate bigram count}} = \frac{6}{6} = 1$
- $p_3 = \frac{\text{clipped trigram count}}{\text{candidate trigram count}} = \frac{5}{5} = 1$
- $p_4 = \frac{\text{clipped four-gram count}}{\text{candidate four-gram count}} = \frac{4}{4} = 1$ (for "the cat is on" and other similar sequences) and other sequences have matching counts as well, but "the cat sat" does not match "the cat is", so $p_4$ actually equals $\frac{4}{5}$ (one less four-gram match) which is $0.8$
Then, we apply the brevity penalty ( $BP$ ): since the candidate sentence has the same length as the reference sentence, $BP = 1$
Next, we compute the BLEU-4 score using the formula: $BLEU-4 = BP \cdot \exp\left(\sum_{n=1}^{4} \frac{1}{4} \log p_n\right) = 1 \cdot \exp\left(\frac{1}{4}(\log 1 + \log 1 + \log 1 + \log 0.8)\right)$
The final output is $1 \cdot \exp\left(\frac{1}{4}(\log 1 + \log 1 + \log 1 + \log 0.8)\right) = 1 \cdot \exp\left(\frac{1}{4}(\log 0.8)\right) = 1 \cdot \exp\left(\frac{1}{4} \cdot -0.2231435513\right) = 1 \cdot \exp(-0.055785388)\approx 0.668

Constraints

If candidate has fewer than 4 tokens, some p_n will be 0 → BLEU = 0
Use geometric mean (equal weights 1/4 each)
Round to 4 decimal places

Learn this in

Evaluation & BenchmarksLLM Study Plan

More from LLM 3: Applications & Evaluation

Beam Search Decoderhard INT8 Quantization Simulatormedium Nucleus (Top-P) Samplingmedium Top-K Samplingeasy Temperature Samplingeasy

Concept primerPREVIEW

Background Knowledge

The BLEU-4 score is a widely used evaluation metric in natural language processing (NLP) to measure the quality of machine translation systems. It calculates the similarity between a candidate translation and one or more reference translations. The score is based on the concept of n-grams, which are sequences of n items (in this case, words) from a given text. The BLEU-4 score uses n-gram precisions for n=1,2,3,4, providing a more comprehensive assessment of the translation quality.

The brevity penalty (BP) is a component of the BLEU-4 score that penalizes candidate translations that are shorter than the reference translations. This is because shorter translations may have a higher n-gram precision simply due to their length, rather than their actual quality. The brevity penalty ensures that the BLEU-4 score rewards translations that are both accurate and complete. The modified n-gram precision is another crucial concept, which calculates the ratio of clipped counts (the number of n-grams in the candidate translation that match the reference translation) to the total number of n-grams in the candidate translation.

The BLEU-4 score formula combines the brevity penalty and the modified n-gram precisions for n=1,2,3,4. If any of the modified n-gram precisions are zero, the BLEU-4 score is set to zero, indicating a poor translation quality. Understanding the components of the BLEU-4 score and how they interact is essential for implementing an accurate calculation.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Tokenize the reference and candidate translations into individual words or n-grams
Calculate the modified n-gram precisions for n=1,2,3,4
Compute the brevity penalty based on the length of the candidate and reference translations
Combine the brevity penalty and modified n-gram precisions to calculate the BLEU-4 score

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results