CHAIR Hallucination Rate

Problem Statement

Given the object words a VLM produced for each image and the ground-truth objects actually present, compute the two CHAIR hallucination rates and list which objects were hallucinated.

Background

CHAIR (Caption Hallucination Assessment with Image Relevance) measures how often a captioner names objects that are not in the image. It has two variants:

$\text{CHAIR}_i = \frac{\#\{\text{hallucinated objects mentioned}\}}{\#\{\text{all objects mentioned}\}}, \qquad \text{CHAIR}_s = \frac{\#\{\text{captions with} \ge 1 \text{ hallucination}\}}{\#\{\text{captions}\}}$

CHAIR_i is instance-level (how noisy is the average mention) and CHAIR_s is sentence-level (what fraction of outputs are contaminated at all). CHAIR_s is always at least as alarming as CHAIR_i, because one bad word condemns a whole caption.

Three details are load-bearing:

1. Synonym mapping. Raw caption words are mapped to canonical MSCOCO categories - "man", "woman", "person" all map to person. Words absent from the map are not object mentions at all and are simply dropped.
2. Deduplicate within a caption. A caption saying "dog" twice mentions one object. Deduplicate after mapping, per caption, so "man" and "woman" in the same sentence count once.
3. Order. The hallucinated-object list you return must be sorted alphabetically, and unique. A raw Python set has no defined iteration order, so sort it before returning - the same trap that makes hallucination-eval scripts produce different output on different runs.

Your Task

Implement:

def chair_metrics(captions, ground_truth, synonym_map):

Return [chair_i, chair_s, hallucinated_objects] where the first two are floats rounded to 4 decimals and the third is a sorted list of the unique canonical object names that were hallucinated anywhere in the dataset.

Input Format

• captions - list of n lists of raw word strings, one list per image
• ground_truth - list of n lists of canonical object names present in that image
• synonym_map - dict mapping a raw word to its canonical object name

Output Format

[float, float, [str, ...]]

Sample

caps = [["a", "man", "riding", "a", "horse"], ["dog", "dog", "frisbee"]]
gt = [["person", "horse"], ["dog"]]
syn = {"man": "person", "woman": "person", "horse": "horse", "dog": "dog", "frisbee": "frisbee"}
print(chair_metrics(caps, gt, syn))

Output:

[0.25, 0.5, ['frisbee']]