Effective Negatives from Sharded Contrastive Batches
Problem Statement
Contrastive learning lives or dies by the number of negatives each example sees. When training data-parallel across devices, whether an example sees negatives from other devices depends on whether embeddings are all-gathered. Compute the number of negatives per positive under both regimes.
Background
Let per_device be the local batch size and world_size the number of devices, so the global batch is G = per_device * world_size. For any anchor image, its one matching text is the positive; every other text in the compared set is a negative.
- With all-gather (CLIP/SigLIP as published): each anchor is compared against the global batch, so it sees G - 1 negatives.
- Without all-gather (naive local loss): each anchor only sees its device's per_device texts, i.e. per_device - 1 negatives.
Your Task
Implement:
def effective_negatives(per_device, world_size):
Return a dict with "global_batch", "neg_with_gather", "neg_local", "gain" where gain = neg_with_gather / neg_local rounded to 4 decimals (gain is 0.0 if neg_local is 0).
Input Format
- per_device (int), world_size (int).
Output Format
- A dict: three ints and one float.
Sample
print(effective_negatives(256, 8))
Output:
{'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}
Example:
print(effective_negatives(256, 8))
{'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}- Calculate the global batch size G by multiplying the local batch size by the number of devices: G=256×8=2048.
- Determine the number of negatives when using all-gather, which is the global batch size minus the anchor's own positive pair: 2048−1=2047.
- Determine the number of negatives in the local regime, which is the local batch size minus the anchor's own positive pair: 256−1=255.
- Compute the gain by dividing the global negatives by the local negatives and rounding to four decimal places: 2047/255≈8.02745→8.0275.
- The final output is
{'global_batch': 2048, 'neg_with_gather': 2047, 'neg_local': 255, 'gain': 8.0275}.
Constraints:
1 <= per_device <= 100000,1 <= world_size <= 4096.neg_with_gather = global_batch - 1,neg_local = per_device - 1.gainrounded to 4 decimals;0.0whenneg_local == 0.
1. Background Knowledge
Contrastive learning trains a model to pull matching pairs (positives) together in embedding space while pushing non-matching pairs (negatives) apart. The quality of the learned representation is heavily influenced by the number of negatives each anchor is compared against: more negatives provide a richer gradient signal and sharper decision boundaries. In vision-language models like CLIP and SigLIP, each image is matched to its corresponding text caption, and every other text in the batch serves as a negative example.
In data-parallel training, the global batch is split across world_size devices, each holding per_device samples. Without communication, a device only computes its local loss, so each anchor sees only per_device - 1 negatives. With all-gather, embeddings from all devices are collected, allowing each anchor to be compared against the full global batch of size G=per_device×world_size, yielding G−1 negatives. This is the regime used in the original CLIP paper and SigLIP.
The gain metric quantifies the benefit of all-gather: it is the ratio of negatives seen with global comparison to those seen locally. A higher gain means the model benefits more from cross-device communication, which is especially pronounced when world_size is large relative to per_device.
2. Algorithm Approach
This is a straightforward arithmetic computation problem. The approach involves:
- Computing the global batch size as the product of per_device and world_size.
- Deriving the two negative counts by subtracting 1 from the relevant batch size (since the positive pair is excluded).
- Computing the gain as a floating-point division, with a guard against division by zero.
- Rounding the gain to 4 decimal places and assembling the result into a dictionary.
No loops, data structures, or iterative logic are needed—just careful handling of the edge case where per_device is 1 (making neg_local zero).
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.