Recall@K and Median Rank Together

1. Background Knowledge

Recall@K is a standard metric in information retrieval and image-text retrieval. For a query with a single relevant item, the item has a rank equal to its 1-based position in the ranked list of candidates. If the relevant item is not present in the list at all, its rank is defined as $\text{len}(\text{list}) + 1$. Recall@K for a single query is 1 if $\text{rank} \le K$, otherwise 0. The overall R@K is the mean of these binary indicators across all queries, expressed as a percentage.

The median rank provides a complementary view. While R@K only tells you whether the relevant item made it into the top-K, the median rank tells you how far the relevant item typically sits in the ranking. For an even number of queries, the median is the arithmetic mean of the two middle ranks after sorting. A lower median rank indicates better retrieval quality.

In vision-language model (VLM) evaluation, these metrics are computed over a held-out test set where each query image (or text) has exactly one ground-truth match. The "ranked-id list" represents the model's ordering of all candidate items by similarity score.

2. Algorithm Approach

This is a single-pass aggregation problem. You iterate over each query once, compute its rank, and accumulate:

• A counter for each $K \in \text{ks}$: increment if $\text{rank} \le K$.
• A list (or running structure) of all ranks for the median computation.

After the loop, divide each counter by the total number of queries to get the percentage, and compute the median from the collected ranks. No sorting of the original rankings is needed since they are already sorted by the model.

3. Step-by-Step Strategy