Zero-Shot Classifier by Prompt Ensembling
Problem Statement
Build a CLIP zero-shot classifier by ensembling several text prompts per class, then classify a batch of image embeddings and report accuracy.
Background
CLIP classifies without any classification head: each class name is written into a handful of prompt templates ("a photo of a {c}.", "a blurry photo of a {c}.", "a sketch of a {c}."), each prompt is encoded, and the class's weight vector is the average of those embeddings. The class weights become the rows of a linear classifier over image embeddings - a head synthesised from text alone.
The order of operations is what makes or breaks it:
- L2-normalise each prompt embedding - so every template contributes equally regardless of its norm
- Average the normalised prompt embeddings for the class
- Re-normalise the average - a mean of unit vectors is not a unit vector, and unless you renormalise, classes whose prompts disagree get a shorter weight vector and are systematically under-predicted
wc​=∥tˉc​∥2​tˉc​​,tˉc​=P1​∑p=1P​∥tc,p​∥2​tc,p​​
Then normalise each image embedding and take the argmax of I @ W.T. Prompt ensembling done this way is worth about a point and a half of ImageNet top-1 over a single template, for zero extra training.
Your Task
Implement:
def zero_shot_classify(image_emb, class_prompt_emb, labels):
Return [preds, accuracy] where preds is the list of predicted class indices (argmax; ties break to the lowest class index) and accuracy is the fraction correct, rounded to 4 decimals.
Input Format
- image_emb - n x d nested list of unnormalised image embeddings
- class_prompt_emb - list of C entries; entry c is a list of P_c unnormalised prompt embeddings of dim d (classes may have different prompt counts)
- labels - list of n ground-truth class indices
Output Format
[[int, ...], float]
Sample
imgs = [[1.0, 0.1], [0.0, 2.0]]
prompts = [[[1.0, 0.0], [0.9, 0.2]], [[0.0, 1.0], [0.1, 0.9]]]
print(zero_shot_classify(imgs, prompts, [0, 1]))
Output:
[[0, 1], 1.0]
Example:
imgs = [[1.0, 0.1], [0.0, 2.0]] prompts = [[[1.0, 0.0], [0.9, 0.2]], [[0.0, 1.0], [0.1, 0.9]]] print(zero_shot_classify(imgs, prompts, [0, 1]))
[[0, 1], 1.0]
Class 0's two unit prompts average to a vector near [0.99, 0.11] which renormalises to a unit vector pointing mostly along x; class 1's points mostly along y. Image 0 leans x and image 1 is pure y, so both are classified correctly: accuracy 1.0.
Constraints:
1 <= n <= 200,1 <= C <= 50,1 <= d <= 64; no vector is the zero vector- Normalise each prompt embedding BEFORE averaging, and re-normalise the class mean AFTER
- Classes may have different numbers of prompts
- Argmax ties break to the lowest class index
- Round accuracy to 4 decimals
1. Background Knowledge
Contrastive Language-Image Pre-training (CLIP) represents a paradigm shift in computer vision by learning visual representations from natural language supervision. Unlike traditional supervised learning, which requires a fixed classification head trained on specific labels, CLIP learns a joint embedding space where images and their corresponding text descriptions are close together. This allows for zero-shot classification, where the model can classify images into categories it has never seen during training, simply by encoding the class names as text.
The core mechanism relies on prompt ensembling. A single text template (e.g., "a photo of a {class}") might not capture the full semantic breadth of a class. By using multiple templates (e.g., "a sketch of a {class}", "a blurry photo of a {class}"), we create a more robust representation. The problem highlights a critical mathematical nuance: simply averaging raw embeddings is suboptimal because embeddings with larger norms dominate the average. Therefore, the standard protocol involves L2-normalizing each individual prompt embedding before averaging, and then re-normalizing the resulting class vector. This ensures that the final class weight vector lies on the unit hypersphere, making the classification equivalent to computing cosine similarity.
In this context, classification is performed via a linear probe. If W is the matrix of normalized class weight vectors and I is the matrix of normalized image embeddings, the similarity scores are computed as S=IWT. The predicted class for each image is the index of the maximum score in its corresponding row. This approach leverages the geometric properties of the embedding space, where cosine similarity serves as the decision metric.
2. Algorithm Approach
The solution follows a deterministic vector algebra pipeline. The approach can be broken down into three main phases: Preprocessing, Weight Construction, and Inference.
- Preprocessing: Normalize all input vectors. This includes every individual prompt embedding and every image embedding. Normalization ensures that the magnitude of the vectors does not influence the similarity calculation, focusing solely on direction (cosine similarity).
- Weight Construction: For each class, aggregate its multiple prompt embeddings. This involves summing the normalized prompt vectors for a class and dividing by the number of prompts to get the mean. Crucially, this mean vector must then be re-normalized to unit length. This results in a single weight vector per class.
- Inference: Compute the dot product between each normalized image embedding and each normalized class weight vector. This can be efficiently done using matrix multiplication. Finally, determine the class with the highest score for each image and compare against ground truth to calculate accuracy.
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.