📘

ROUGE-N Score

MediumEvaluation Metrics 📚 LLM 3: Applications & Evaluation

Compute the ROUGE-N recall score.

ROUGE-N measures n-gram overlap between candidate and reference: $\text{ROUGE-N} = \frac{\sum_{\text{n-gram}} \min(\text{count}_{cand}(\text{n-gram}), \text{count}_{ref}(\text{n-gram}))}{\sum_{\text{n-gram}} \text{count}_{ref}(\text{n-gram})}$

Note: ROUGE uses recall (denominator is reference count).

Input:

Line 1: n (n-gram size)
Line 2: reference text
Line 3: candidate text

Output: ROUGE-N recall score, rounded to 4 decimal places.

Example:

Input:

1
the cat sat on the mat
the cat sat

Output:

0.5000

Reasoning:

The n-gram size is 1, so we split the reference text "the cat sat on the mat" into 1-grams: ["the", "cat", "sat", "on", "the", "mat"].
We also split the candidate text "the cat sat" into 1-grams: ["the", "cat", "sat"].
For each 1-gram, we calculate the minimum count between the candidate and reference texts: $min(1, 2)$ for "the", $min(1, 1)$ for "cat", $min(1, 1)$ for "sat", $min(0, 1)$ for "on", and $min(0, 1)$ for "mat".
We then calculate the ROUGE-1 recall score using the formula: $\frac{1 + 1 + 1}{2 + 1 + 1 + 1 + 1} = \frac{3}{6} = 0.5$ , which is rounded to 4 decimal places as $0.5000$ .

Constraints:

1 <= n <= 4
Case-sensitive
Round to 4 decimal places
If reference has no n-grams, output 0.0

More from LLM 3: Applications & Evaluation

Beam Search DecoderHard INT8 Quantization SimulatorMedium Nucleus (Top-P) SamplingMedium Top-K SamplingEasy Temperature SamplingEasy

Background Knowledge

The ROUGE-N score is a measure used to evaluate the quality of text summarization and machine translation systems. It calculates the overlap between the n-grams (sequences of n items) in the candidate text and the reference text. The n-gram concept is crucial here, as it allows us to compare texts at different levels of granularity. For example, unigrams (1-grams) compare individual words, while bigrams (2-grams) compare pairs of words.

The recall aspect of the ROUGE-N score is important, as it measures the proportion of n-grams in the reference text that are also present in the candidate text. This is different from precision, which would measure the proportion of n-grams in the candidate text that are also present in the reference text. The formula for ROUGE-N recall score is given by $\frac{\sum_{\text{n-gram}} \min(\text{count}_{cand}(\text{n-gram}), \text{count}_{ref}(\text{n-gram}))}{\sum_{\text{n-gram}} \text{count}_{ref}(\text{n-gram})}$ , where $\text{count}_{cand}(\text{n-gram})$ and $\text{count}_{ref}(\text{n-gram})$ represent the counts of a particular n-gram in the candidate and reference texts, respectively.

To understand this problem, one should be familiar with basic text processing techniques, such as tokenization (splitting text into individual words or tokens) and n-gram extraction. Additionally, understanding the concept of recall and its importance in evaluation metrics is essential.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Extract n-grams from both the reference and candidate texts
Calculate the count of each n-gram in both texts
Calculate the minimum count for each n-gram between the two texts
Calculate the sum of these minimum counts and the sum of n-gram counts in the reference text
Use these sums to calculate the ROUGE-N recall score

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.