Zero-Shot Classifier by Prompt Ensembling

Problem Statement

Build a CLIP zero-shot classifier by ensembling several text prompts per class, then classify a batch of image embeddings and report accuracy.

Background

CLIP classifies without any classification head: each class name is written into a handful of prompt templates ("a photo of a {c}.", "a blurry photo of a {c}.", "a sketch of a {c}."), each prompt is encoded, and the class's weight vector is the average of those embeddings. The class weights become the rows of a linear classifier over image embeddings - a head synthesised from text alone.

The order of operations is what makes or breaks it:

1. L2-normalise each prompt embedding - so every template contributes equally regardless of its norm
2. Average the normalised prompt embeddings for the class
3. Re-normalise the average - a mean of unit vectors is not a unit vector, and unless you renormalise, classes whose prompts disagree get a shorter weight vector and are systematically under-predicted

$w_c = \frac{\bar{t}_c}{\lVert \bar t_c \rVert_2}, \qquad \bar t_c = \frac{1}{P}\sum_{p=1}^{P} \frac{t_{c,p}}{\lVert t_{c,p} \rVert_2}$

Then normalise each image embedding and take the argmax of I @ W.T. Prompt ensembling done this way is worth about a point and a half of ImageNet top-1 over a single template, for zero extra training.

Your Task

Implement:

def zero_shot_classify(image_emb, class_prompt_emb, labels):

Return [preds, accuracy] where preds is the list of predicted class indices (argmax; ties break to the lowest class index) and accuracy is the fraction correct, rounded to 4 decimals.

Input Format

• image_emb - n x d nested list of unnormalised image embeddings
• class_prompt_emb - list of C entries; entry c is a list of P_c unnormalised prompt embeddings of dim d (classes may have different prompt counts)
• labels - list of n ground-truth class indices

Output Format

[[int, ...], float]

Sample

imgs = [[1.0, 0.1], [0.0, 2.0]]
prompts = [[[1.0, 0.0], [0.9, 0.2]], [[0.0, 1.0], [0.1, 0.9]]]
print(zero_shot_classify(imgs, prompts, [0, 1]))

Output:

[[0, 1], 1.0]