Zero-Shot Classifier by Prompt Ensembling

mediumPRORetrieval & Zero-Shot TransferVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Build a CLIP zero-shot classifier by ensembling several text prompts per class, then classify a batch of image embeddings and report accuracy.

Background

CLIP classifies without any classification head: each class name is written into a handful of prompt templates ("a photo of a {c}.", "a blurry photo of a {c}.", "a sketch of a {c}."), each prompt is encoded, and the class's weight vector is the average of those embeddings. The class weights become the rows of a linear classifier over image embeddings - a head synthesised from text alone.

The order of operations is what makes or breaks it:

L2-normalise each prompt embedding - so every template contributes equally regardless of its norm
Average the normalised prompt embeddings for the class
Re-normalise the average - a mean of unit vectors is not a unit vector, and unless you renormalise, classes whose prompts disagree get a shorter weight vector and are systematically under-predicted

$w_c = \frac{\bar{t}_c}{\lVert \bar t_c \rVert_2}, \qquad \bar t_c = \frac{1}{P}\sum_{p=1}^{P} \frac{t_{c,p}}{\lVert t_{c,p} \rVert_2}$

Then normalise each image embedding and take the argmax of I @ W.T. Prompt ensembling done this way is worth about a point and a half of ImageNet top-1 over a single template, for zero extra training.

Your Task

Implement:

def zero_shot_classify(image_emb, class_prompt_emb, labels):

Return [preds, accuracy] where preds is the list of predicted class indices (argmax; ties break to the lowest class index) and accuracy is the fraction correct, rounded to 4 decimals.

Input Format

image_emb - n x d nested list of unnormalised image embeddings
class_prompt_emb - list of C entries; entry c is a list of P_c unnormalised prompt embeddings of dim d (classes may have different prompt counts)
labels - list of n ground-truth class indices

Output Format

[[int, ...], float]

Sample

imgs = [[1.0, 0.1], [0.0, 2.0]]
prompts = [[[1.0, 0.0], [0.9, 0.2]], [[0.0, 1.0], [0.1, 0.9]]]
print(zero_shot_classify(imgs, prompts, [0, 1]))

Output:

[[0, 1], 1.0]

Example

Input

imgs = [[1.0, 0.1], [0.0, 2.0]]
prompts = [[[1.0, 0.0], [0.9, 0.2]], [[0.0, 1.0], [0.1, 0.9]]]
print(zero_shot_classify(imgs, prompts, [0, 1]))

Output

[[0, 1], 1.0]

Reasoning

Class 0's two unit prompts average to a vector near [0.99, 0.11] which renormalises to a unit vector pointing mostly along x; class 1's points mostly along y. Image 0 leans x and image 1 is pure y, so both are classified correctly: accuracy 1.0.

Constraints

1 <= n <= 200, 1 <= C <= 50, 1 <= d <= 64; no vector is the zero vector
Normalise each prompt embedding BEFORE averaging, and re-normalise the class mean AFTER
Classes may have different numbers of prompts
Argmax ties break to the lowest class index
Round accuracy to 4 decimals

Learn this in

Multimodal LLMsLLM Study Plan Contrastive Learning & CLIPVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

Contrastive Language-Image Pre-training (CLIP) represents a paradigm shift in computer vision by learning visual representations from natural language supervision. Unlike traditional supervised learning, which requires a fixed classification head trained on specific labels, CLIP learns a joint embedding space where images and their corresponding text descriptions are close together. This allows for zero-shot classification, where the model can classify images into categories it has never seen during training, simply by encoding the class names as text.

The core mechanism relies on prompt ensembling. A single text template (e.g., "a photo of a {class}") might not capture the full semantic breadth of a class. By using multiple templates (e.g., "a sketch of a {class}", "a blurry photo of a {class}"), we create a more robust representation. The problem highlights a critical mathematical nuance: simply averaging raw embeddings is suboptimal because embeddings with larger norms dominate the average. Therefore, the standard protocol involves L2-normalizing each individual prompt embedding before averaging, and then re-normalizing the resulting class vector. This ensures that the final class weight vector lies on the unit hypersphere, making the classification equivalent to computing cosine similarity.

In this context, classification is performed via a linear probe. If $W$ is the matrix of normalized class weight vectors and $I$ is the matrix of normalized image embeddings, the similarity scores are computed as $S = I W^T$ . The predicted class for each image is the index of the maximum score in its corresponding row. This approach leverages the geometric properties of the embedding space, where cosine similarity serves as the decision metric.

2. Algorithm Approach

The solution follows a deterministic vector algebra pipeline. The approach can be broken down into three main phases: Preprocessing, Weight Construction, and Inference.

Preprocessing: Normalize all input vectors. This includes every individual prompt embedding and every image embedding. Normalization ensures that the magnitude of the vectors does not influence the similarity calculation, focusing solely on direction (cosine similarity).
Weight Construction: For each class, aggregate its multiple prompt embeddings. This involves summing the normalized prompt vectors for a class and dividing by the number of prompts to get the mean. Crucially, this mean vector must then be re-normalized to unit length. This results in a single weight vector per class.
Inference: Compute the dot product between each normalized image embedding and each normalized class weight vector. This can be efficiently done using matrix multiplication. Finally, determine the class with the highest score for each image and compare against ground truth to calculate accuracy.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results