ROUGE Score

mediumPROText Generation & AdvancedNLP 3: Advanced

Implement the ROUGE-N metric for evaluating text summarization.

ROUGE-N measures n-gram overlap between a generated summary and a reference summary.

$\text{ROUGE-N} = \frac{\sum_{\text{n-gram} \in \text{ref}} \min(\text{count}_{\text{gen}}(\text{n-gram}), \text{count}_{\text{ref}}(\text{n-gram}))}{\sum_{\text{n-gram} \in \text{ref}} \text{count}_{\text{ref}}(\text{n-gram})}$

In simpler terms: sum of clipped overlapping n-gram counts divided by total reference n-gram count.

Function: rouge_n(n, generated, reference)

n: n-gram size (integer)
generated: generated summary (string of space-separated lowercase words)
reference: reference summary (string of space-separated lowercase words)

Returns: the ROUGE-N recall score as a float rounded to 4 decimal places.

Example

Input

1
the cat sat on the mat
the cat sat on a rug

Output

0.6667

Reasoning

Inputs: n is 1, so the units being compared are single words. The generated summary is "the cat sat on the mat" and the reference is "the cat sat on a rug", each six words long.
Reference counts: the, cat, sat, on, a and rug each appear once, so the reference holds $6$ unigrams. This total is the denominator.
Generated counts: "the" appears twice, while cat, sat, on and mat appear once each.
Clipping: for each reference word, credit is $\min(\text{count}_{\text{gen}}, \text{count}_{\text{ref}})$ . "the" earns $\min(2, 1) = 1$ , not 2, because the reference only contains one "the". cat, sat and on earn 1 each, while a and rug earn 0. "mat" is not in the reference, so it never enters the sum.
Score: $\frac{1+1+1+1+0+0}{6} = \frac{4}{6} \approx 0.6667$ , rounded to 4 decimal places, giving 0.6667.
Notice: with $n = 2$ the same pair scores $\frac{3}{5} = 0.6$ . Only "the cat", "cat sat" and "sat on" match among the reference's 5 bigrams, so longer n-grams punish word-order changes harder.

Constraints

N-grams are word-level
Clip generated counts to reference counts
ROUGE is recall-oriented (denominator = reference n-gram count)
If reference has no n-grams (too short), output 0.0
Round to 4 decimal places

Learn this in

Text Generation & SummarizationNLP Study Plan

More from NLP 3: Advanced

Temperature Samplereasy ROUGE-1 Score Calculatoreasy Context Window Findereasy Keyword Answer Extractoreasy Levenshtein Distanceeasy

Concept primerPREVIEW

Background Knowledge

The ROUGE score is a measure used to evaluate the quality of text summarization and machine translation systems. It calculates the overlap between the generated summary and a reference summary, typically using n-gram matching. An n-gram is a sequence of $n$ items from a given text or speech. For example, in the sentence "the cat sat", the 1-grams are ["the", "cat", "sat"], the 2-grams are ["the cat", "cat sat"], and the 3-grams are ["the cat sat"].

The ROUGE-N metric specifically measures the overlap of n-grams of size $n$ between the generated and reference summaries. This is done by counting the number of n-grams that appear in both summaries and dividing it by the total number of n-grams in the reference summary. The result is a recall score that indicates how well the generated summary covers the content of the reference summary. The formula for ROUGE-N is given by $\text{ROUGE-N} = \frac{\sum_{\text{n-gram} \in \text{ref}} \min(\text{count}_{\text{gen}}(\text{n-gram}), \text{count}_{\text{ref}}(\text{n-gram}))}{\sum_{\text{n-gram} \in \text{ref}} \text{count}_{\text{ref}}(\text{n-gram})}$

Understanding n-grams and how to calculate them is crucial for implementing the ROUGE-N metric. Additionally, being familiar with string processing and counting techniques in programming will be helpful. The ROUGE score is an important metric in Natural Language Processing (NLP), as it provides a way to objectively evaluate the quality of text summarization systems.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Read the input and parse the generated and reference summaries into lists of words.
Generate all n-grams of size $n$ from both summaries.
Count the occurrences of each n-gram in both summaries.
Calculate the ROUGE-N score using the formula provided.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.