ROUGE-1 Score Calculator

easyFREEText Generation & AdvancedNLP 3: Advanced

Calculate the ROUGE-1 F1 score between a generated summary and a reference summary.

ROUGE-1 measures unigram overlap. A word that appears $a$ times in the generated summary and $b$ times in the reference contributes $\min(a, b)$ matching unigrams:

Precision = (matching unigrams) / (unigrams in generated)
Recall = (matching unigrams) / (unigrams in reference)
F1 = 2 * Precision * Recall / (Precision + Recall)

All comparisons are case-insensitive. Round the F1 score to 4 decimal places.

Input:

gen: the generated summary (a string)
ref: the reference summary (a string)

Return the F1 score as a float.

Example

Input

the cat sat on the mat
the cat is on the mat

Output

0.8333

Reasoning

Inputs: gen is the generated summary and ref the reference; the unigrams are each text's whitespace-separated words, compared ignoring case. Both sample sentences have 6 words.
Word counts: generated has the ×2, cat, sat, on, mat; reference has the ×2, cat, is, on, mat.
Matches are clipped: each word counts $\min(a, b)$ times, where $a$ and $b$ are its counts in the two texts. the gives $\min(2,2)=2$ ; cat, on, mat give 1 each; sat and is give 0. Total $2+1+1+1=5$ .
Scores: $P = 5/6$ , $R = 5/6$ , so $F_1 = \frac{2 \cdot \frac{5}{6} \cdot \frac{5}{6}}{\frac{5}{6}+\frac{5}{6}} = \frac{5}{6} \approx 0.8333$ , the output 0.8333.
Notice: when $P = R$ , $F_1$ equals that shared value. Clipping stops repetition from inflating the score: generated "The THE the" against reference "the" has only 1 match, so $P = 1/3$ , $R = 1$ and $F_1 = 0.5$ . With no shared words (0.0) both are 0.

Constraints

Case-insensitive comparison
Use word-level unigrams (split on whitespace)
If precision + recall = 0, F1 = 0.0
Round F1 to 4 decimal places

More from NLP 3: Advanced

Temperature Samplereasy Context Window Findereasy Keyword Answer Extractoreasy Levenshtein Distanceeasy Word-Level Translatoreasy

Concept primer

Background Knowledge

The ROUGE-1 score is a metric used to evaluate the quality of text summaries. It measures the overlap between the generated summary and a reference summary at the unigram level, which refers to individual words or tokens. The precision of the generated summary is the ratio of matching unigrams to the total number of unigrams in the generated summary. The recall is the ratio of matching unigrams to the total number of unigrams in the reference summary. The F1 score is the harmonic mean of precision and recall, providing a balanced measure of both.

In the context of Natural Language Processing (NLP), the ROUGE-1 score is particularly useful for evaluating the performance of text summarization models. It helps to assess how well a generated summary captures the key information present in the reference summary. The case-insensitive comparison ensures that the evaluation is not affected by differences in capitalization between the generated and reference summaries.

Understanding the concepts of tokenization, set operations, and statistical measures (such as precision, recall, and F1 score) is essential for calculating the ROUGE-1 score. Tokenization involves splitting text into individual words or tokens, while set operations are used to find the intersection (matching unigrams) and union of the generated and reference summaries. The F1 score calculation involves simple arithmetic operations to compute the harmonic mean of precision and recall.

Algorithm/Approach

The general approach to solving this problem involves the following steps:

Tokenize the generated and reference summaries into individual words or unigrams.
Compute the precision and recall by finding the matching unigrams and dividing by the total number of unigrams in the generated and reference summaries, respectively.
Calculate the F1 score as the harmonic mean of precision and recall.

This approach can be implemented using basic programming constructs, such as loops, conditional statements, and arithmetic operations.

Step-by-Step Strategy

To implement the solution, follow these steps:

Read the generated and reference summaries from the input.
Tokenize the summaries into individual words or unigrams, and convert them to lowercase for case-insensitive comparison.
Find the matching unigrams between the generated and reference summaries.
Compute the precision and recall using the matching unigrams and the total number of unigrams in the generated and reference summaries.
Calculate the F1 score as the harmonic mean of precision and recall, and round it to 4 decimal places.

Common Pitfalls

When implementing the solution, watch out for the following:

Ensure that the tokenization is done correctly, and punctuation is handled properly.
Use case-insensitive comparison to avoid incorrect results due to differences in capitalization.
Handle edge cases, such as empty summaries or summaries with no matching unigrams.

Time & Space Complexity

The time complexity of the solution is expected to be O(n + m), where n and m are the lengths of the generated and reference summaries, respectively. This is because the tokenization and set operations can be performed in linear time. The space complexity is also O(n + m), as we need to store the tokenized summaries and the matching unigrams.

solution.py

Python 3.13

Test results