NER F1 Calculator

mediumPRONamed Entity RecognitionNLP 2: Applications

Compute entity-level precision, recall, and F1 score for NER evaluation.

An entity is considered correct only if both the span (start and end positions) AND the entity type match exactly.

Input:

predicted: list of predicted entities, each a (start, end, type) tuple
gold: list of gold/true entities, each a (start, end, type) tuple

Output: A string with three values separated by spaces: "precision recall f1" (each rounded to 4 decimal places, e.g. 0.6667 1.0 0.8). If precision or recall is undefined (0/0), output 0.0 for that metric and F1.

Example

Input

3
0 1 PER
4 6 LOC
8 8 ORG
2
0 1 PER
4 6 LOC

Output

0.6667 1.0 0.8

Reasoning

Inputs: predicted holds 3 entities — $(0,1,\text{PER})$ , $(4,6,\text{LOC})$ , $(8,8,\text{ORG})$ — and gold holds 2: $(0,1,\text{PER})$ , $(4,6,\text{LOC})$ . Each entity is a token span (start, end, inclusive) plus a type label.
Matching: a prediction counts only if its start, end and type all equal some gold entity. The PER span 0–1 and the LOC span 4–6 agree exactly; the ORG at token 8 has no gold partner, so it is a false positive. Correct $= 2$ .
Precision asks "how many predictions were right": $2/3 \approx 0.6667$ . Recall asks "how many gold entities were found": $2/2 = 1.0$ .
F1: use the exact fractions, $\frac{2 \cdot \frac{2}{3} \cdot 1}{\frac{2}{3} + 1} = \frac{4/3}{5/3} = 0.8$ . Rounding only happens at the end.
Output: the string 0.6667 1.0 0.8, values rounded to 4 decimal places with trailing zeros dropped. Notice the extra ORG guess cost precision but not recall; a wrong type on a correct span (say $(0,1,\text{ORG})$ against a gold $(0,1,\text{PER})$ ) would cost both.

Constraints

An entity match requires exact start, end, AND type match
Precision = correct / predicted_count
Recall = correct / gold_count
F1 = 2 * P * R / (P + R), or 0.0 if P + R = 0
Round to 4 decimal places

Learn this in

Named Entity RecognitionNLP Study Plan

More from NLP 2: Applications

BIO Entity Span Extractoreasy Simple Entity Taggereasy Sentiment Word Countereasy Keyword Classifiereasy One-Hot Sequence Encodereasy

Concept primerPREVIEW

Background Knowledge

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing named entities in unstructured text into predefined categories such as names, locations, organizations, etc. The performance of NER models is typically evaluated using metrics like precision, recall, and F1 score. In the context of NER, these metrics are calculated at the entity level, meaning that an entity is considered correct only if both its span (start and end positions) and its type match exactly with a gold entity.

The precision of a NER model refers to the ratio of true positives (correctly identified entities) to the sum of true positives and false positives (incorrectly identified entities). Recall is the ratio of true positives to the sum of true positives and false negatives (missed entities). The F1 score is the harmonic mean of precision and recall, providing a balanced measure of both. These metrics are essential for evaluating the performance of NER models, as they help in understanding the model's ability to correctly identify entities without over-predicting or under-predicting.

Understanding the concepts of true positives, false positives, and false negatives is crucial for calculating these metrics. A true positive is an entity that is correctly identified by the model (both span and type match). A false positive is an entity that is incorrectly identified (either the span or the type does not match any gold entity), and a false negative is an entity that is missed by the model (a gold entity that does not have a matching predicted entity).

Algorithm/Approach

The general approach to solving this problem involves reading the input data, which includes the predicted entities and the gold entities, and then calculating the true positives, false positives, and false negatives based on the given conditions. This requires iterating through both sets of entities and comparing their spans and types to determine matches. The precision, recall, and F1 score can then be calculated using the formulas: $Precision = \frac{TP}{TP + FP}$ , $Recall = \frac{TP}{TP + FN}$ , and $F1 = \frac{2 \cdot Precision \cdot Recall}{Precision + Recall}$ , where $TP$ is the number of true positives, $FP$ is the number of false positives, and $FN$ is the number of false negatives.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results