CHAIR-i and CHAIR-s Hallucination Rates

mediumPROEfficient Adaptation & Hallucination MetricsVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

CHAIR measures caption hallucination two ways: per-instance (what fraction of mentioned objects are wrong) and per-sentence (what fraction of captions contain any hallucination). Compute both over a set of captions.

Background

For each caption you get the set of objects the model mentioned and the set of objects actually present (the ground truth). An object is hallucinated if it was mentioned but is not present. Then:

$\text{CHAIR}_i = \frac{\sum_c |\text{hallucinated}_c|}{\sum_c |\text{mentioned}_c|}, \qquad \text{CHAIR}_s = \frac{\#\{c : |\text{hallucinated}_c| > 0\}}{\#\text{captions}}$

Lower is better for both. A caption that mentions nothing contributes 0 to both numerators (and 0 mentions to the CHAIR-i denominator).

Your Task

Implement:

def chair(mentioned, present):

mentioned[c], present[c]: lists of object names (treat as sets) for caption c.

Return a dict with "chair_i" and "chair_s", each rounded to 4 decimals. If total mentions is 0, chair_i is 0.0.

Input Format

mentioned: list of lists of object names.
present: list of lists of object names, same length.

Output Format

A dict of two floats.

Sample

m = [["dog", "cat"], ["car"]]
p = [["dog"], ["car", "tree"]]
print(chair(m, p))

Output:

{'chair_i': 0.3333, 'chair_s': 0.5}

Example

Input

m = [["dog", "cat"], ["car"]]
p = [["dog"], ["car", "tree"]]
print(chair(m, p))

Output

{'chair_i': 0.3333, 'chair_s': 0.5}

Reasoning

Caption 0 mentions {dog,cat}, present {dog}: cat hallucinated (1 of 2). Caption 1 mentions {car}, present {car,tree}: 0 hallucinated. CHAIR-i = 1/3 = 0.3333; CHAIR-s = 1 of 2 captions = 0.5.

Constraints

len(mentioned) == len(present), 1 <= num_captions <= 100000.
Treat each list as a set (ignore duplicates within a caption).
A hallucinated object is mentioned but not present.
Round both to 4 decimals; chair_i is 0.0 when there are no mentions.

Learn this in

Benchmarks, Deployment & the FrontierVLM Study Plan Instruction Tuning & the LLaVA ParadigmVLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium POPE Object Hallucination Accuracyeasy

Concept primerPREVIEW

1. Background Knowledge

Hallucination in Vision-Language Models (VLMs) refers to the generation of objects or attributes that are not present in the input image. Because VLMs are autoregressive, they can "confabulate" details to make captions sound coherent, even when the visual evidence is absent. Quantifying this behavior is essential for evaluating model reliability in safety-critical applications like medical imaging or autonomous driving.

CHAIR (Caption Hallucination Assessment with Image Reference) is a standard metric for this purpose. It distinguishes between two granularities:

CHAIR $_i$ (Instance-level): Measures the density of errors. If a model mentions 10 objects and 2 are wrong, the error rate is 20%. This penalizes models that hallucinate frequently, regardless of how many objects they mention in total.
CHAIR $_s$ (Sentence-level): Measures the frequency of errors. If 1 out of 5 captions contains any hallucination, the rate is 20%. This penalizes models that fail to produce even a single clean caption.

The distinction is crucial: a model might have a low CHAIR $_s$ (most captions are perfect) but a high CHAIR $_i$ (when it does hallucinate, it hallucinates many objects). Conversely, a model with low CHAIR $_i$ might still have a high CHAIR $_s$ if it rarely mentions objects but occasionally gets one wrong.

2. Algorithm Approach

This is a set-difference aggregation problem. The core logic involves:

Per-caption analysis: For each caption, determine which mentioned objects are hallucinated. This is a set difference operation: $\text{hallucinated}_c = \text{mentioned}_c \setminus \text{present}_c$ .
Aggregation:

For CHAIR $_i$ , sum the sizes of all hallucinated sets and divide by the sum of all mentioned set sizes.
For CHAIR $_s$ , count how many captions have a non-empty hallucinated set and divide by the total number of captions.

The approach is linear in the number of captions and objects. There is no need for complex data structures beyond sets for efficient membership testing.

3. Step-by-Step Strategy

Initialize accumulators:

total_mentions = 0 (for CHAIR $_i$ denominator)
total_hallucinations = 0 (for CHAIR $_i$ numerator)
hallucinated_captions = 0 (for CHAIR $_s$ numerator)
total_captions = len(mentioned)

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results