Vision-Language Models
VLM 1: Vision-Language Models
Fifteen hands-on exercises covering the arithmetic that actually runs a vision-language model: patch tokenisation and image-token budgets, the CLIP and SigLIP contrastive objectives, retrieval and zero-shot transfer, bridge modules from linear projectors to the Perceiver Resampler, and the metrics for efficient adaptation and hallucination. Every problem takes its arrays as input - no vision encoder is ever run.
50
problems
5
topics
15
free
01
Bridge Modules: Projectors, Q-Former & Cross-Attention
10 problems- 1Q-Former Output Shape and Parameter CountFREEEasy
- 2MLP Projector Parameter CountPROEasy
- 3Attention Pooling to a Single TokenPROEasy
- 4Scaled Dot-Product Cross-AttentionFREEMedium
- 5Multi-Head Cross-Attention OutputPROMedium
- 6Gated Cross-Attention (Flamingo tanh Gate)PROMedium
- 7Learned Query Compression RatioPROMedium
- 8RMSNorm Before the ProjectorPROMedium
- 9Perceiver Resampler Forward PassFREEHard
- 10One Perceiver Resampler BlockPROHard
02
Contrastive Objectives: CLIP and SigLIP
10 problems- 11L2-Normalize an EmbeddingFREEEasy
- 12Contrastive Accuracy at the DiagonalPROEasy
- 13CLIP Contrastive (InfoNCE) LossFREEMedium
- 14SigLIP Pairwise Sigmoid LossPROMedium
- 15Symmetric CLIP Loss with a Learnable TemperaturePROMedium
- 16SigLIP Loss with the Learnable BiasPROMedium
- 17Effective Negatives from Sharded Contrastive BatchesPROMedium
- 18Modality Gap Between Image and Text EmbeddingsPROMedium
- 19Softmax vs Sigmoid: Batch Dependence of the Match ProbabilityFREEHard
- 20Hard Negative Mining in a Contrastive BatchPROHard
03
Efficient Adaptation & Hallucination Metrics
10 problemsSign up free to see all 50 problems
Create a free account to see the full VLM 1: Vision-Language Models list, track your progress, and save problems to lists.
Sign up free