POPE Precision, Recall, and F1

1. Background Knowledge

In Vision-Language Models (VLMs), hallucination refers to the model generating content that is not present in the input image. The POPE (Polling-based Object Probing Evaluation) benchmark is designed to detect this by asking binary yes/no questions about object presence. Because POPE splits are balanced (roughly 50% "yes" and 50% "no" ground truths), a model that always answers "yes" would achieve ~50% accuracy but have terrible precision. This is why precision is the critical metric: it measures how often the model is correct when it claims something is present.

Precision, Recall, and F1 are standard classification metrics. With "yes" as the positive class:

• Precision = $\frac{\text{TP}}{\text{TP} + \text{FP}}$ (of all "yes" answers, how many were correct?)
• Recall = $\frac{\text{TP}}{\text{TP} + \text{FN}}$ (of all actual "yes" cases, how many did we catch?)
• F1 = $\frac{2 \cdot P \cdot R}{P + R}$ (harmonic mean, balancing both)

When a denominator is zero (e.g., the model never says "yes"), the metric is defined as $0.0$ to avoid division by zero.

2. Algorithm Approach

This is a straightforward confusion matrix aggregation problem. The approach is:

1. Normalize all strings to lowercase for case-insensitive comparison.
2. Iterate through the paired lists once, counting TP, FP, and FN.
3. Compute the three metrics using the formulas above, handling zero-division cases.
4. Round each result to 4 decimal places and return as a dictionary.

No sorting, searching, or complex data structures are needed—just a single pass with counters.

3. Step-by-Step Strategy

• Normalize inputs: Convert each element in answers and truths to lowercase. You can do this inline during iteration or pre-process the lists.
• Initialize counters: Set tp = fp = fn = 0.
• Single pass: Loop over indices (or use zip):
• If answer == "yes" and truth == "yes": increment tp
• If answer == "yes" and truth == "no": increment fp
• If answer == "no" and truth == "yes": increment fn
• (The "no/no" case is TN and is not needed for these metrics)
• Compute metrics:
• precision = tp / (tp + fp) if tp + fp > 0, else 0.0
• recall = tp / (tp + fn) if tp + fn > 0, else 0.0
• f1 = 2 * precision * recall / (precision + recall) if precision + recall > 0, else 0.0
• Round and return: Apply round(value, 4) to each metric and return {"precision":..., "recall":..., "f1":...}.

4. Common Pitfalls

• Case sensitivity: The problem states comparison is case-insensitive. Forgetting to call .lower() on both answers and truths will cause mismatches like "Yes" vs "yes".
• Division by zero: If the model never answers "yes" (tp + fp == 0) or never misses a "yes" (tp + fn == 0), you must return 0.0 instead of crashing. The same applies to F1 when both P and R are zero.
• Rounding: The problem requires rounding to 4 decimal places. Using round(x, 4) is correct; truncating or using fewer decimals will fail hidden tests.
• Confusing FP and FN: FP is "false positive" (said yes, truth is no)—this is the hallucination count. FN is "false negative" (said no, truth is yes). Mixing these up swaps precision and recall.
• Assuming only "yes"/"no": While the problem guarantees yes/no strings, defensive coding (e.g., stripping whitespace) can help if inputs have trailing spaces.

5. Time & Space Complexity

• Time Complexity: $O(n)$, where $n = \text{len}(answers)$. A single linear pass through the lists with constant-time operations per element.
• Space Complexity: $O(1)$ extra space (excluding input storage). Only a fixed number of integer counters and float variables are used regardless of input size. If you pre-normalize the lists, that becomes $O(n)$ space, but in-place normalization during iteration keeps it $O(1)$.