NER F1 Calculator
Compute entity-level precision, recall, and F1 score for NER evaluation.
An entity is considered correct only if both the span (start and end positions) AND the entity type match exactly.
Input format:
- Line 1: Number of predicted entities P
- Lines 2 to P+1: start end type (predicted entities)
- Line P+2: Number of gold/true entities G
- Lines P+3 to P+G+2: start end type (gold entities)
Output: Three values on one line: precision recall f1 (each rounded to 4 decimal places). If precision or recall is undefined (0/0), output 0.0 for that metric and F1.
Example:
3 0 1 PER 4 6 LOC 8 8 ORG 2 0 1 PER 4 6 LOC
0.6667 1.0 0.8
Predicted entities: (0,1,PER), (4,6,LOC), (8,8,ORG) Gold entities: (0,1,PER), (4,6,LOC)
Matches: (0,1,PER) and (4,6,LOC) both match => 2 correct
Precision: 2/3 = 0.6667 (2 correct out of 3 predicted) Recall: 2/2 = 1.0 (2 correct out of 2 gold) F1: 2 * 0.6667 * 1.0 / (0.6667 + 1.0) = 1.3333 / 1.6667 = 0.8
Constraints:
- An entity match requires exact start, end, AND type match
- Precision = correct / predicted_count
- Recall = correct / gold_count
- F1 = 2 * P * R / (P + R), or 0.0 if P + R = 0
- Round to 4 decimal places
Background Knowledge
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing named entities in unstructured text into predefined categories such as names, locations, organizations, etc. The performance of NER models is typically evaluated using metrics like precision, recall, and F1 score. In the context of NER, these metrics are calculated at the entity level, meaning that an entity is considered correct only if both its span (start and end positions) and its type match exactly with a gold entity.
The precision of a NER model refers to the ratio of true positives (correctly identified entities) to the sum of true positives and false positives (incorrectly identified entities). Recall is the ratio of true positives to the sum of true positives and false negatives (missed entities). The F1 score is the harmonic mean of precision and recall, providing a balanced measure of both. These metrics are essential for evaluating the performance of NER models, as they help in understanding the model's ability to correctly identify entities without over-predicting or under-predicting.
Understanding the concepts of true positives, false positives, and false negatives is crucial for calculating these metrics. A true positive is an entity that is correctly identified by the model (both span and type match). A false positive is an entity that is incorrectly identified (either the span or the type does not match any gold entity), and a false negative is an entity that is missed by the model (a gold entity that does not have a matching predicted entity).
Algorithm/Approach
The general approach to solving this problem involves reading the input data, which includes the predicted entities and the gold entities, and then calculating the true positives, false positives, and false negatives based on the given conditions. This requires iterating through both sets of entities and comparing their spans and types to determine matches. The precision, recall, and F1 score can then be calculated using the formulas: Precision=TP+FPTP​, Recall=TP+FNTP​, and F1=Precision+Recall2⋅Precision⋅Recall​, where TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.