POPE Object Hallucination Accuracy
Problem Statement
POPE probes object hallucination with yes/no questions ("Is there a chair in the image?"). Score a batch of the model's yes/no answers against the ground truth.
Background
Each item has a ground-truth label ("yes" if the object is present, "no" if not) and the model's answer. Accuracy is the fraction of matching answers:
acc=N#{i:answeri​=truthi​}​
A model that hallucinates says "yes" to absent objects, tanking accuracy on the "no" questions.
Your Task
Implement:
def pope_accuracy(answers, truths):
Return the accuracy as a float rounded to 4 decimals. Comparison is case-insensitive and ignores surrounding whitespace.
Input Format
- answers: list of strings (the model's answers).
- truths: list of strings (ground truth), same length.
Output Format
- A float rounded to 4 decimals.
Sample
print(pope_accuracy(["yes", "No", "yes"], ["yes", "no", "no"]))
Output:
0.6667
Example:
print(pope_accuracy(["yes", "No", "yes"], ["yes", "no", "no"]))
0.6667
- Normalize the inputs by stripping whitespace and converting to lowercase to ensure case-insensitive comparison: the answers become
["yes", "no", "yes"]and the truths become["yes", "no", "no"]. - Compare each pair of normalized strings to determine correctness:
- Pair 1:
"yes"vs"yes"→ Match - Pair 2:
"no"vs"no"→ Match - Pair 3:
"yes"vs"no"→ Mismatch
- Pair 1:
- Count the total number of matches: 2 out of 3 total items are correct.
- Calculate the accuracy by dividing the number of correct answers by the total number of items: acc=32​≈0.66666...
- Round the result to 4 decimal places as required: 0.6667.
- The final output is 0.6667
Constraints:
len(answers) == len(truths),1 <= N <= 100000.- Compare case-insensitively after stripping whitespace.
- Round to 4 decimals.
1. Background Knowledge
Object hallucination is a well-known failure mode in Vision-Language Models (VLMs). A model may confidently assert the presence of an object that is not in the image (e.g., answering "yes" to "Is there a chair?" when no chair exists). The POPE (Polling-based Object Probing Evaluation) benchmark specifically targets this by posing binary yes/no questions about object presence and measuring how often the model's answer matches ground truth.
The core metric here is accuracy, defined as the fraction of predictions that exactly match the ground-truth label:
acc=N#{i:answeri​=truthi​}​where N is the total number of questions. A model that systematically hallucinates will score poorly on the "no" subset because it answers "yes" to absent objects.
In practice, model outputs are often noisy: they may contain extra whitespace, mixed casing ("Yes", "YES", " no "), or trailing newlines. Robust evaluation requires normalizing both the prediction and the ground truth before comparison.
2. Algorithm Approach
This is a straightforward element-wise comparison problem. The pattern is:
- Normalize each answer and truth string (strip whitespace, convert to lowercase).
- Compare each normalized pair for equality.
- Count the number of matches.
- Divide by the total number of items to get the accuracy fraction.
- Round the result to 4 decimal places.
No data structures beyond a simple counter are needed. The entire computation is a single linear pass over the two lists.
3. Step-by-Step Strategy
- Validate inputs: Confirm that answers and truths have the same length. If either list is empty, decide on a sensible return value (e.g., 0.0 or raise an error).
- Normalize each string: For every element, call .strip() to remove leading/trailing whitespace, then .lower() to unify casing. This makes " Yes " and "yes" equivalent.
- Iterate and count: Loop over the indices (or use zip) and increment a counter whenever the normalized answer equals the normalized truth.
- Compute accuracy: Divide the match count by the total number of items N. Use floating-point division.
- Round and return: Apply round(value, 4) to produce the required 4-decimal output.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.