Effective Negatives from Sharded Contrastive Batches

mediumPROContrastive Objectives: CLIP and SigLIPVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Contrastive learning lives or dies by the number of negatives each example sees. When training data-parallel across devices, whether an example sees negatives from other devices depends on whether embeddings are all-gathered. Compute the number of negatives per positive under both regimes.

Background

Let per_device be the local batch size and world_size the number of devices, so the global batch is G = per_device * world_size. For any anchor image, its one matching text is the positive; every other text in the compared set is a negative.

With all-gather (CLIP/SigLIP as published): each anchor is compared against the global batch, so it sees G - 1 negatives.
Without all-gather (naive local loss): each anchor only sees its device's per_device texts, i.e. per_device - 1 negatives.

Your Task

Implement:

def effective_negatives(per_device, world_size):

Return a dict with "global_batch", "neg_with_gather", "neg_local", "gain" where gain = neg_with_gather / neg_local rounded to 4 decimals (gain is 0.0 if neg_local is 0).

Input Format

per_device (int), world_size (int).

Output Format

A dict: three ints and one float.

Sample

print(effective_negatives(256, 8))

Output:

{'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}

Example

Input

print(effective_negatives(256, 8))

Output

{'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}

Reasoning

Calculate the global batch size $G$ by multiplying the local batch size by the number of devices: $G = 256 \times 8 = 2048$ .
Determine the number of negatives when using all-gather, which is the global batch size minus the anchor's own positive pair: $2048 - 1 = 2047$ .
Determine the number of negatives in the local regime, which is the local batch size minus the anchor's own positive pair: $256 - 1 = 255$ .
Compute the gain by dividing the global negatives by the local negatives and rounding to four decimal places: $2047 / 255 \approx 8.02745 \rightarrow 8.0275$ .
The final output is {'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}.

Constraints

1 <= per_device <= 100000, 1 <= world_size <= 4096.
neg_with_gather = global_batch - 1, neg_local = per_device - 1.
gain rounded to 4 decimals; 0.0 when neg_local == 0.

Learn this in

Multimodal LLMsLLM Study Plan Question Answering & Information RetrievalNLP Study Plan Contrastive Learning & CLIPVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

Contrastive learning trains a model to pull matching pairs (positives) together in embedding space while pushing non-matching pairs (negatives) apart. The quality of the learned representation is heavily influenced by the number of negatives each anchor is compared against: more negatives provide a richer gradient signal and sharper decision boundaries. In vision-language models like CLIP and SigLIP, each image is matched to its corresponding text caption, and every other text in the batch serves as a negative example.

In data-parallel training, the global batch is split across world_size devices, each holding per_device samples. Without communication, a device only computes its local loss, so each anchor sees only per_device - 1 negatives. With all-gather, embeddings from all devices are collected, allowing each anchor to be compared against the full global batch of size $G = \text{per\_device} \times \text{world\_size}$ , yielding $G - 1$ negatives. This is the regime used in the original CLIP paper and SigLIP.

The gain metric quantifies the benefit of all-gather: it is the ratio of negatives seen with global comparison to those seen locally. A higher gain means the model benefits more from cross-device communication, which is especially pronounced when world_size is large relative to per_device.

2. Algorithm Approach

This is a straightforward arithmetic computation problem. The approach involves:

Computing the global batch size as the product of per_device and world_size.
Deriving the two negative counts by subtracting 1 from the relevant batch size (since the positive pair is excluded).
Computing the gain as a floating-point division, with a guard against division by zero.
Rounding the gain to 4 decimal places and assembling the result into a dictionary.

No loops, data structures, or iterative logic are needed—just careful handling of the edge case where per_device is 1 (making neg_local zero).

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.