Vision Language Models Study Plan
A comprehensive 10-week curriculum covering Vision Language Models from CLIP and vision encoders to LLaVA, LoRA fine-tuning, and deploying frontier VLMs.
Recommended Study Path
Prerequisites
Foundations Study Plan
- Python Foundations
- NumPy & Linear Algebra
- Probability & Calculus
Complete the Foundations study plan first →
Foundations
Weeks 1-2
- Ch 1: Intro to VLMs
- Ch 2: Vision Encoders
ViT, CLIP, SigLIP, vision backbones
Core Training
Weeks 3-8
- Ch 3-6: Architecture & Training
- Ch 7-8: RLHF, LoRA & Fine-tuning
LLaVA, CLIP, DPO, QLoRA, adapters
Applications
Weeks 9-10
- Ch 9: Reasoning & Applications
- Ch 10: Deployment & Frontier
Visual agents, GPT-4o, Gemini, serving
All Chapters
Introduction to Vision-Language Models
What VLMs are, the vision-language gap, evolution from VQA to GPT-4V, architecture overview, key capabilities, and the landscape of modern VLMs.
Vision Encoders
CNN feature extractors, Vision Transformers, CLIP visual encoder, SigLIP, InternViT, DINOv2, and choosing the right vision backbone.
Language Model Backbone
Decoder-only vs encoder-decoder architectures, multimodal tokenization, attention mechanisms in VLMs, popular LLM backbones, and scaling laws.
Vision-Language Alignment
Bridging the representation gap with linear projection, Q-Former, perceiver resampler, MLP projectors, dynamic resolution, and architecture comparisons.
Contrastive Learning & CLIP
Contrastive learning foundations, InfoNCE loss, CLIP training pipeline, zero-shot transfer, ALIGN and Florence variants, and limitations.
Instruction Tuning & the LLaVA Paradigm
Visual instruction tuning, GPT-4 data generation, two-stage training, LLaVA-1.5/NeXT, conversation formats, SFT data, and evaluation.
Training Recipes: Pretraining, SFT & RLHF
Pretraining objectives, web-scale data, supervised fine-tuning, RLHF for VLMs, DPO, hallucination mitigation, and training infrastructure.
Parameter-Efficient Fine-Tuning
LoRA, QLoRA, adapter tuning, prefix tuning, practical VLM fine-tuning strategies, and adapter merging for deployment.
Multimodal Reasoning & Applications
Visual chain-of-thought, grounding, document understanding, video VLMs, multi-image reasoning, agentic VLMs, and spatial understanding.
Benchmarks, Deployment & the Frontier
VLM benchmarks, quantization, serving infrastructure, inference optimization, frontier models, open challenges, and future directions.
Practice Problem Sets
Sharpen your skills with coding challenges and system design problems.
Curriculum designed to take you from vision-language fundamentals to production-ready multimodal systems.