BLEU Score (Unigram)
Compute the unigram BLEU precision score.
BLEU unigram precision is the fraction of tokens in the candidate that appear in the reference, with clipped counts: p1=∑wcountcand(w)∑wmin(countcand(w),countref(w))
Also apply a brevity penalty: BP={1e1−r/cif c>rif c≤r
where c = candidate length, r = reference length.
BLEU = BP × p_1
Input:
- Line 1: reference (space-separated tokens)
- Line 2: candidate (space-separated tokens)
Output: BLEU score, rounded to 4 decimal places.
Example:
the cat sat on the mat the cat sat
0.6065
- First, we count the occurrences of each token in the reference and candidate: reference = [the: 2, cat: 1, sat: 1, on: 1, mat: 1], candidate = [the: 1, cat: 1, sat: 1]
- Then, we calculate the unigram precision p1=countcand(the)+countcand(cat)+countcand(sat)min(countcand(the),countref(the))+min(countcand(cat),countref(cat))+min(countcand(sat),countref(sat))=1+1+11+1+1=33=1
- Next, we calculate the brevity penalty: since c=3 (candidate length) and r=6 (reference length), c≤r, so BP=e1−r/c=e1−6/3=e1−2=e−1≈0.6065
- The final BLEU score is the product of BP and p1: BLEU=BP×p1≈0.6065×1=0.6065
Constraints:
- Case-sensitive comparison
- Clipped counts: min of candidate count and reference count
- Round to 4 decimal places
More from LLM 3: Applications & Evaluation
Background Knowledge
The BLEU score is a widely used evaluation metric in natural language processing (NLP) to measure the quality of machine-generated text, such as translations or summaries. It is based on the idea of comparing the generated text (candidate) with one or more reference texts. The BLEU score calculates the similarity between the candidate and reference texts by comparing the frequency of n-grams (sequences of n items) in both texts. In this problem, we are focusing on unigram BLEU, which means we are only considering individual words (1-grams) in our comparison.
The unigram BLEU precision is calculated as the ratio of the number of words in the candidate that also appear in the reference to the total number of words in the candidate. This is done with clipped counts, meaning that for each word, we only count the minimum number of occurrences in both the candidate and the reference. This prevents words that appear more frequently in the candidate than in the reference from being over-counted. Additionally, a brevity penalty is applied to penalize candidates that are shorter than the reference, as these may not capture all the information in the reference.
The brevity penalty is calculated based on the lengths of the candidate and reference texts. If the candidate is longer than the reference, no penalty is applied. If the candidate is shorter, the penalty is calculated using an exponential function that depends on the ratio of the candidate length to the reference length. The final BLEU score is the product of the unigram precision and the brevity penalty, providing a balanced measure of both the accuracy and completeness of the generated text.
Algorithm/Approach
The approach to solving this problem involves several key steps: tokenizing the input texts, calculating the clipped counts of words in both texts, computing the unigram precision, calculating the brevity penalty, and finally combining these components to obtain the BLEU score. This problem can be seen as an application of basic string processing and statistical analysis techniques, where the focus is on comparing and analyzing the frequency of words in two different texts.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.