📘

ROUGE Score

MediumText Generation & Advanced 📚 NLP 3: Advanced

Implement the ROUGE-N metric for evaluating text summarization.

ROUGE-N measures n-gram overlap between a generated summary and a reference summary.

$\text{ROUGE-N} = \frac{\sum_{\text{n-gram} \in \text{ref}} \min(\text{count}_{\text{gen}}(\text{n-gram}), \text{count}_{\text{ref}}(\text{n-gram}))}{\sum_{\text{n-gram} \in \text{ref}} \text{count}_{\text{ref}}(\text{n-gram})}$

In simpler terms: sum of clipped overlapping n-gram counts divided by total reference n-gram count.

Input format:

Line 1: N (n-gram size, integer)
Line 2: Generated summary (space-separated lowercase words)
Line 3: Reference summary (space-separated lowercase words)

Output: ROUGE-N recall score rounded to 4 decimal places.

Example:

Input:

1
the cat sat on the mat
the cat sat on a rug

Output:

0.6667

Reasoning:

ROUGE-1 (unigram):

Reference unigrams: the(1), cat(1), sat(1), on(1), a(1), rug(1) => total = 6 Generated unigrams: the(2), cat(1), sat(1), on(1), mat(1)

Overlap (clipped by ref count):

the: min(2, 1) = 1
cat: min(1, 1) = 1
sat: min(1, 1) = 1
on: min(1, 1) = 1
a: min(0, 1) = 0
rug: min(0, 1) = 0

Sum of clipped: 4 ROUGE-1 = 4/6 = 0.6667

Constraints:

N-grams are word-level
Clip generated counts to reference counts
ROUGE is recall-oriented (denominator = reference n-gram count)
If reference has no n-grams (too short), output 0.0
Round to 4 decimal places

More from NLP 3: Advanced

Temperature SamplerEasy ROUGE-1 Score CalculatorEasy Context Window FinderEasy Keyword Answer ExtractorEasy Levenshtein DistanceEasy

Background Knowledge

The ROUGE score is a measure used to evaluate the quality of text summarization and machine translation systems. It calculates the overlap between the generated summary and a reference summary, typically using n-gram matching. An n-gram is a sequence of $n$ items from a given text or speech. For example, in the sentence "the cat sat", the 1-grams are ["the", "cat", "sat"], the 2-grams are ["the cat", "cat sat"], and the 3-grams are ["the cat sat"].

The ROUGE-N metric specifically measures the overlap of n-grams of size $n$ between the generated and reference summaries. This is done by counting the number of n-grams that appear in both summaries and dividing it by the total number of n-grams in the reference summary. The result is a recall score that indicates how well the generated summary covers the content of the reference summary. The formula for ROUGE-N is given by $\text{ROUGE-N} = \frac{\sum_{\text{n-gram} \in \text{ref}} \min(\text{count}_{\text{gen}}(\text{n-gram}), \text{count}_{\text{ref}}(\text{n-gram}))}{\sum_{\text{n-gram} \in \text{ref}} \text{count}_{\text{ref}}(\text{n-gram})}$

Understanding n-grams and how to calculate them is crucial for implementing the ROUGE-N metric. Additionally, being familiar with string processing and counting techniques in programming will be helpful. The ROUGE score is an important metric in Natural Language Processing (NLP), as it provides a way to objectively evaluate the quality of text summarization systems.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Read the input and parse the generated and reference summaries into lists of words.
Generate all n-grams of size $n$ from both summaries.
Count the occurrences of each n-gram in both summaries.
Calculate the ROUGE-N score using the formula provided.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.