Temperature-Scaled Zero-Shot Confidence

mediumPRORetrieval & Zero-Shot TransferVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Zero-shot logits are cosine similarities in [-1, 1], so a raw softmax over them is nearly flat and useless as a confidence. Apply CLIP's logit scale (1/tau), then report the predicted class and its calibrated probability.

Background

CLIP multiplies cosine similarities by a learned logit_scale (equivalently divides by a temperature tau) before the softmax:

$p_c = \text{softmax}\!\left(\frac{\text{sim}_c}{\tau}\right)_c$

A small tau (large logit scale, ~100 at convergence) sharpens the distribution so the top class gets a meaningful probability. Return the argmax class and its probability. Use a numerically stable softmax (subtract the max).

Your Task

Implement:

def zero_shot_confidence(sims, tau):

Return a tuple (pred_index, probability) where probability is rounded to 4 decimals. Ties in the argmax go to the smallest index.

Input Format

sims: list of cosine similarities (one per class).
tau (float): temperature, tau > 0.

Output Format

A tuple (int, float).

Sample

print(zero_shot_confidence([0.3, 0.2, 0.1], 0.01))

Output:

(0, 1.0)

Example

Input

print(zero_shot_confidence([0.3, 0.2, 0.1], 0.01))

Output

(0, 1.0)

Reasoning

Scale the similarities: Divide each cosine similarity by the temperature $\tau = 0.01$ to sharpen the distribution, resulting in logits $z = [0.3/0.01, 0.2/0.01, 0.1/0.01] = [30, 20, 10]$ .
Stabilize for exponentiation: Subtract the maximum logit ( $30$ ) from each value to prevent numerical overflow, yielding shifted values $[0, -10, -20]$ .
Compute unnormalized probabilities: Apply the exponential function to the shifted values, giving $e^0 = 1$ , $e^{-10} \approx 4.54 \times 10^{-5}$ , and $e^{-20} \approx 2.06 \times 10^{-9}$ .
Normalize to get probabilities: Divide each exponential value by their sum ( $\approx 1.0000454$ ), resulting in probabilities $[0.9999546, 4.54 \times 10^{-5}, 2.06 \times 10^{-9}]$ .
Determine prediction and confidence: The maximum probability occurs at index $0$ with a value of $\approx 0.9999546$ ; rounding this to 4 decimal places yields $1.0$ .
The final output is (0, 1.0)

Constraints

1 <= len(sims) <= 10000, tau > 0.
Divide by tau before the softmax; use a stable softmax.
Argmax ties go to the smallest index; probability rounded to 4 decimals.

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

In zero-shot classification with Vision-Language Models like CLIP, the model computes a cosine similarity between a visual embedding and a set of text embeddings (one per candidate class). These raw similarities lie in the range $[-1, 1]$ . If you apply a standard softmax directly to these values, the resulting probabilities are nearly uniform because the differences between classes are tiny relative to the scale of the input. For example, a difference of $0.1$ in cosine similarity is negligible when the values themselves are around $0.5$ .

To fix this, CLIP introduces a logit scale parameter, often denoted as $\frac{1}{\tau}$ , where $\tau$ is the temperature. The learned logit scale at convergence is typically around $100$ , meaning $\tau \approx 0.01$ . By dividing the similarities by this small $\tau$ , the differences between classes are amplified by a factor of $100$ . This sharpens the softmax distribution, allowing the model to express high confidence in the most similar class. The calibrated probability for class $c$ is computed as:

p_c = \frac{e^{\text{sim}_c / \tau}}{\sum_{j} e^{\text{sim}_j / \tau}}

This process is known as temperature scaling. A small $\tau$ makes the distribution "peaky," while a large $\tau$ makes it "flat." In practice, $\tau$ is learned during training to optimize a contrastive loss, but for inference, it is fixed.

2. Algorithm Approach

The problem requires implementing a numerically stable softmax followed by an argmax operation. The core algorithmic pattern is:

Scale: Divide each similarity score by the temperature $\tau$ .
Stabilize: Subtract the maximum scaled value from all scaled values to prevent numerical overflow in the exponential function.
Exponentiate: Compute $e^{x_i}$ for each stabilized value.
Normalize: Divide each exponentiated value by the sum of all exponentiated values to get probabilities.
Predict: Find the index of the maximum probability (which corresponds to the maximum similarity) and return that index along with its probability.

Note that the argmax of the softmax is identical to the argmax of the original similarities, since softmax is a monotonic transformation. However, you must compute the full probability distribution to report the calibrated confidence.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results