Contrastive Accuracy at the Diagonal
Problem Statement
In a CLIP batch of N paired image-text examples, the correct match for image i is text i — the diagonal of the similarity matrix. Measure the image-to-text retrieval accuracy: the fraction of rows whose argmax lands on the diagonal.
Background
Given the N x N logits matrix S (row = image, column = text), the image-to-text prediction for row i is argmax_j S[i][j]. It is correct iff that argmax equals i. Break ties by the smallest column index (numpy's argmax default). Accuracy is the count of correct rows divided by N.
Your Task
Implement:
def contrastive_accuracy(logits):
Return the image-to-text accuracy as a float rounded to 4 decimals.
Input Format
- logits: N x N nested list.
Output Format
- A float rounded to 4 decimals.
Sample
print(contrastive_accuracy([[2.0, 1.0], [0.5, 3.0]]))
Output:
1.0
Example:
print(contrastive_accuracy([[2.0, 1.0], [0.5, 3.0]]))
1.0
- Convert the input nested list into a 2×2 matrix S to represent the similarity scores between images and texts: S=[2.00.51.03.0]
- Determine the predicted text index for each image by finding the column index of the maximum value in each row (resolving ties by the smallest index, though none exist here):
- Row 0: max(2.0,1.0)=2.0 at index 0.
- Row 1: max(0.5,3.0)=3.0 at index 1.
- The prediction vector is [0,1].
- Compare these predictions against the ground truth diagonal indices [0,1] to count correct matches:
- Image 0 predicted text 0 (Correct).
- Image 1 predicted text 1 (Correct).
- Total correct predictions = 2.
- Calculate the accuracy by dividing the number of correct predictions by the total number of examples N=2: Accuracy=22=1.0
- The final output is 1.0
Constraints:
1 <= N <= 1000; the matrix is square.- Ties in a row go to the smallest column index.
- Accuracy = correct_rows / N, rounded to 4 decimals.
1. Background Knowledge
Contrastive learning is a paradigm where a model learns by distinguishing between positive pairs (matching items) and negative pairs (non-matching items). In CLIP (Contrastive Language–Image Pre-training), each image is paired with its correct text caption, and the model learns to maximize the similarity between matched pairs while minimizing similarity between mismatched pairs.
The core computation produces an N×N logits matrix S, where S[i][j] represents the similarity score between image i and text j. The diagonal entries S[i][i] correspond to the correct (positive) pairs. During training, a cross-entropy loss is applied row-wise: for each image i, the model is trained so that S[i][i] is the highest value in row i relative to all other S[i][j] values.
Retrieval accuracy is the standard evaluation metric. For image-to-text retrieval, you check whether the text with the highest similarity score for each image is indeed its correct caption. This is equivalent to checking whether the argmax of each row falls on the diagonal of the similarity matrix.
2. Algorithm Approach
This is a straightforward row-wise argmax problem. For each row i in the N×N matrix:
- Find the column index j∗=argmaxjS[i][j].
- Check if j∗=i (i.e., the argmax lands on the diagonal).
- Count the number of rows where this condition holds.
- Divide by N to get the accuracy fraction.
The key insight is that you are performing an independent argmax operation on each row and comparing the result to the row index. No sorting or full matrix operations are needed beyond the per-row maximum search.
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.