Prompt-Ensemble Zero-Shot Classifier Weights

mediumPRORetrieval & Zero-Shot TransferVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

CLIP's zero-shot accuracy jumps when each class is described by many prompt templates whose embeddings are averaged into a single classifier weight. Build those per-class weights from the template embeddings.

Background

For class c with template embeddings e_{c,1}, ..., e_{c,T} (each already L2-normalized), the ensembled classifier weight is the mean of the normalized templates, then re-normalized:

$w_c = \frac{\bar{e}_c}{\lVert \bar{e}_c \rVert_2}, \qquad \bar{e}_c = \frac{1}{T}\sum_{t} e_{c,t}$

This is exactly what OpenAI's zero-shot code does with its 80 ImageNet templates. The re-normalization matters: averaging unit vectors gives a shorter vector, and the final classifier must sit back on the unit sphere so logits stay comparable across classes.

Your Task

Implement:

def ensemble_weights(class_templates):

class_templates: list over classes; each element is a list of template embeddings (each a list of floats).

Return the (num_classes, D) weight matrix as a nested list rounded to 4 decimals.

Input Format

class_templates: num_classes lists, each of T_c embeddings of dimension D.

Output Format

A (num_classes, D) nested list rounded to 4 decimals.

Sample

ct = [[[1.0, 0.0], [0.0, 1.0]]]
print(ensemble_weights(ct))

Output:

[[0.7071, 0.7071]]

Example

Input

ct = [[[1.0, 0.0], [0.0, 1.0]]]
print(ensemble_weights(ct))

Output

[[0.7071, 0.7071]]

Reasoning

Compute the mean embedding: For the single class, average the two template vectors $[1.0, 0.0]$ and $[0.0, 1.0]$ component-wise to get $\bar{e} = \left[\frac{1.0+0.0}{2}, \frac{0.0+1.0}{2}\right] = [0.5, 0.5]$ .
Calculate the L2 norm: Determine the magnitude of the mean vector to prepare for re-normalization: $\lVert \bar{e} \rVert_2 = \sqrt{0.5^2 + 0.5^2} = \sqrt{0.25 + 0.25} = \sqrt{0.5} \approx 0.7071$ .
Re-normalize the vector: Divide the mean vector by its norm to project it back onto the unit sphere, ensuring the classifier weight has a magnitude of 1: $w = \frac{[0.5, 0.5]}{0.7071} \approx [0.7071, 0.7071]$ .
Round to 4 decimals: Apply the required precision formatting to the resulting weights, yielding $[0.7071, 0.7071]$ .
The final output is [[0.7071, 0.7071]]

Constraints

1 <= num_classes <= 1000, 1 <= T_c, 1 <= D <= 1024.
Average the templates per class, then L2-normalize (guard a zero mean by leaving it zero).
Round every entry to 4 decimals; avoid -0.0.

Learn this in

Contrastive Learning & CLIPVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

Zero-shot classification with Vision-Language Models (VLMs) like CLIP works by projecting both images and text prompts into a shared embedding space. For each class, you generate multiple text prompts (e.g., "a photo of a {class}", "a painting of a {class}"), encode them, and use the resulting vectors as classifier weights. This avoids needing labeled image data for the target classes.

Prompt ensembling improves robustness by averaging embeddings from multiple prompt templates. If class $c$ has $T$ template embeddings $e_{c,1}, \dots, e_{c,T}$ , each already L2-normalized (unit length), the raw ensemble is the arithmetic mean:

$\bar{e}_c = \frac{1}{T}\sum_{t=1}^{T} e_{c,t}$

Averaging unit vectors generally produces a vector with norm strictly less than 1 (unless all templates are identical). Since CLIP computes similarity via dot product (or cosine similarity), the classifier weights must lie on the unit sphere so that logits are comparable across classes. Therefore, the final weight is re-normalized:

$w_c = \frac{\bar{e}_c}{\lVert \bar{e}_c \rVert_2}$

This is precisely what OpenAI's official zero-shot evaluation code does with its 80 ImageNet prompt templates.

2. Algorithm Approach

The problem reduces to a straightforward vector arithmetic pipeline applied independently per class:

For each class, compute the element-wise mean of its template embeddings.
Compute the L2 norm of that mean vector.
Divide every component by the norm to re-normalize.
Round each component to 4 decimal places.

No iterative optimization, matrix inversion, or complex data structures are needed. The core operation is a reduction (sum/mean) followed by a normalization step.

3. Step-by-Step Strategy

Iterate over classes: Loop through class_templates, where each element is a list of $T_c$ embeddings of dimension $D$ .
Compute the mean vector: For each dimension $d \in \{0, \dots, D-1\}$ , sum the $d$ -th component across all $T_c$ templates and divide by $T_c$ .
Compute the L2 norm: Take the square root of the sum of squared components of the mean vector: $\lVert \bar{e}_c \rVert_2 = \sqrt{\sum_{d=0}^{D-1} \bar{e}_{c,d}^2}$
Normalize: Divide each component of the mean vector by the computed norm.
Round: Apply round(value, 4) to each component.
Collect results: Append the normalized, rounded row to the output list.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.