POPE Object Hallucination Accuracy

easyPROEfficient Adaptation & Hallucination MetricsVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

POPE probes object hallucination with yes/no questions ("Is there a chair in the image?"). Score a batch of the model's yes/no answers against the ground truth.

Background

Each item has a ground-truth label ("yes" if the object is present, "no" if not) and the model's answer. Accuracy is the fraction of matching answers:

$\text{acc} = \frac{\#\{i : \text{answer}_i = \text{truth}_i\}}{N}$

A model that hallucinates says "yes" to absent objects, tanking accuracy on the "no" questions.

Your Task

Implement:

def pope_accuracy(answers, truths):

Return the accuracy as a float rounded to 4 decimals. Comparison is case-insensitive and ignores surrounding whitespace.

Input Format

answers: list of strings (the model's answers).
truths: list of strings (ground truth), same length.

Output Format

A float rounded to 4 decimals.

Sample

print(pope_accuracy(["yes", "No", "yes"], ["yes", "no", "no"]))

Output:

0.6667

Example

Input

print(pope_accuracy(["yes", "No", "yes"], ["yes", "no", "no"]))

Output

0.6667

Reasoning

Normalize the inputs by stripping whitespace and converting to lowercase to ensure case-insensitive comparison: the answers become ["yes", "no", "yes"] and the truths become ["yes", "no", "no"].
Compare each pair of normalized strings to determine correctness:
- Pair 1: "yes" vs "yes" → Match
- Pair 2: "no" vs "no" → Match
- Pair 3: "yes" vs "no" → Mismatch
Count the total number of matches: $2$ out of $3$ total items are correct.
Calculate the accuracy by dividing the number of correct answers by the total number of items: $\text{acc} = \frac{2}{3} \approx 0.66666...$
Round the result to 4 decimal places as required: $0.6667$ .
The final output is 0.6667

Constraints

len(answers) == len(truths), 1 <= N <= 100000.
Compare case-insensitively after stripping whitespace.
Round to 4 decimals.

Learn this in

Benchmarks, Deployment & the FrontierVLM Study Plan Instruction Tuning & the LLaVA ParadigmVLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

Object hallucination is a well-known failure mode in Vision-Language Models (VLMs). A model may confidently assert the presence of an object that is not in the image (e.g., answering "yes" to "Is there a chair?" when no chair exists). The POPE (Polling-based Object Probing Evaluation) benchmark specifically targets this by posing binary yes/no questions about object presence and measuring how often the model's answer matches ground truth.

The core metric here is accuracy, defined as the fraction of predictions that exactly match the ground-truth label:

\text{acc} = \frac{\#\{i : \text{answer}_i = \text{truth}_i\}}{N}

where $N$ is the total number of questions. A model that systematically hallucinates will score poorly on the "no" subset because it answers "yes" to absent objects.

In practice, model outputs are often noisy: they may contain extra whitespace, mixed casing ("Yes", "YES", " no "), or trailing newlines. Robust evaluation requires normalizing both the prediction and the ground truth before comparison.

2. Algorithm Approach

This is a straightforward element-wise comparison problem. The pattern is:

Normalize each answer and truth string (strip whitespace, convert to lowercase).
Compare each normalized pair for equality.
Count the number of matches.
Divide by the total number of items to get the accuracy fraction.
Round the result to 4 decimal places.

No data structures beyond a simple counter are needed. The entire computation is a single linear pass over the two lists.

3. Step-by-Step Strategy

Validate inputs: Confirm that answers and truths have the same length. If either list is empty, decide on a sensible return value (e.g., 0.0 or raise an error).
Normalize each string: For every element, call .strip() to remove leading/trailing whitespace, then .lower() to unify casing. This makes " Yes " and "yes" equivalent.
Iterate and count: Loop over the indices (or use zip) and increment a counter whenever the normalized answer equals the normalized truth.
Compute accuracy: Divide the match count by the total number of items $N$ . Use floating-point division.
Round and return: Apply round(value, 4) to produce the required 4-decimal output.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results