Modality Gap Between Image and Text Embeddings

mediumPROContrastive Objectives: CLIP and SigLIPVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Even after contrastive training, image and text embeddings occupy two separated cones on the hypersphere — the "modality gap." Quantify it as the Euclidean distance between the two centroids of the L2-normalized embeddings.

Background

Given a batch of image embeddings and text embeddings, first L2-normalize each vector (contrastive models operate on the unit sphere). The modality gap is the distance between the mean image vector and the mean text vector:

$\Delta = \left\lVert \frac{1}{N}\sum_i \hat{u}_i - \frac{1}{M}\sum_j \hat{v}_j \right\rVert_2$

Liang et al. (2022) showed this gap is created at initialization and persists through training; shrinking it can change downstream behavior.

Your Task

Implement:

def modality_gap(image_embs, text_embs):

Return the gap as a float rounded to 4 decimals.

Input Format

image_embs: N x D nested list.
text_embs: M x D nested list.

Output Format

A float rounded to 4 decimals.

Sample

print(modality_gap([[1.0, 0.0]], [[0.0, 1.0]]))

Output:

1.4142

Example

Input

print(modality_gap([[1.0, 0.0]], [[0.0, 1.0]]))

Output

1.4142

Reasoning

L2 Normalization: Each embedding vector is projected onto the unit sphere to simulate contrastive model behavior. The image vector $[1.0, 0.0]$ has a norm of $\sqrt{1^2 + 0^2} = 1$ , so it remains $\hat{u}_1 = [1.0, 0.0]$ . The text vector $[0.0, 1.0]$ has a norm of $\sqrt{0^2 + 1^2} = 1$ , so it remains $\hat{v}_1 = [0.0, 1.0]$ .
Centroid Calculation: The mean vector for each modality is computed. With only one sample per modality, the image centroid is $\bar{u} = [1.0, 0.0]$ and the text centroid is $\bar{v} = [0.0, 1.0]$ .
Difference Vector: The vector difference between the centroids is calculated to determine the separation: $\bar{u} - \bar{v} = [1.0 - 0.0, 0.0 - 1.0] = [1.0, -1.0]$ .
Euclidean Distance: The modality gap is the L2 norm of this difference vector: $\Delta = \sqrt{1.0^2 + (-1.0)^2} = \sqrt{1 + 1} = \sqrt{2} \approx 1.41421356$ .
The final output is 1.4142

Constraints

1 <= N, M <= 2000, 1 <= D <= 1024.
L2-normalize every vector before averaging (skip zero vectors: leave them as zeros).
Return the Euclidean distance between the two centroids, rounded to 4 decimals.

Learn this in

Advanced NLP TopicsNLP Study Plan Introduction to Vision-Language ModelsVLM Study Plan Vision EncodersVLM Study Plan Vision-Language AlignmentVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

In vision-language models like CLIP and SigLIP, images and text are encoded into a shared embedding space and trained with a contrastive objective (e.g., InfoNCE). A well-documented phenomenon, first characterized by Liang et al. (2022), is the modality gap: even after convergence, the image and text embeddings do not overlap on the hypersphere but instead occupy two distinct, separated cones. This gap is largely determined at initialization and persists throughout training, influencing retrieval accuracy and downstream transfer.

The standard way to quantify this gap is to compute the Euclidean distance between the two centroids (mean vectors) of the L2-normalized embeddings. Because contrastive models operate on the unit hypersphere, every embedding vector must first be normalized to have unit $\ell_2$ norm. The gap is then:

\Delta = \left\lVert \frac{1}{N}\sum_{i=1}^{N} \hat{u}_i - \frac{1}{M}\sum_{j=1}^{M} \hat{v}_j \right\rVert_2

where $\hat{u}_i$ and $\hat{v}_j$ are the normalized image and text vectors, respectively. Note that $N$ and $M$ can differ; the means are computed independently for each modality.

2. Algorithm Approach

This is a straightforward vector arithmetic problem with three conceptual stages:

L2-normalize every row vector in both input matrices.
Compute the centroid (row-wise mean) of each normalized matrix.
Take the Euclidean norm of the difference between the two centroids.

No iterative optimization or complex data structure is needed. The entire computation is a small number of element-wise operations over $O((N+M) \cdot D)$ elements.

3. Step-by-Step Strategy

Normalize each vector: For a vector $x \in \mathbb{R}^D$ , compute $\hat{x} = x / \|x\|_2$ . Guard against zero-norm vectors (rare but possible) to avoid division by zero.
Compute the image centroid: Average all $N$ normalized image vectors element-wise to get a single $D$ -dimensional vector $\bar{u}$ .
Compute the text centroid: Similarly average all $M$ normalized text vectors to get $\bar{v}$ .
Compute the gap: Calculate $\|\bar{u} - \bar{v}\|_2 = \sqrt{\sum_{d=1}^{D}(\bar{u}_d - \bar{v}_d)^2}$ .
Round the result to 4 decimal places and return as a float.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.