Loading...
A comprehensive 10-week curriculum covering Vision Language Models from CLIP and vision encoders to LLaVA, LoRA fine-tuning, and deploying frontier VLMs.
Foundations Study Plan
Complete the Foundations study plan first →
Weeks 1-2
ViT, CLIP, SigLIP, vision backbones
Weeks 3-8
LLaVA, CLIP, DPO, QLoRA, adapters
Weeks 9-10
Visual agents, GPT-4o, Gemini, serving
What VLMs are, the vision-language gap, evolution from VQA to GPT-4V, architecture overview, key capabilities, and the landscape of modern VLMs.
CNN feature extractors, Vision Transformers, CLIP visual encoder, SigLIP, InternViT, DINOv2, and choosing the right vision backbone.
Decoder-only vs encoder-decoder architectures, multimodal tokenization, attention mechanisms in VLMs, popular LLM backbones, and scaling laws.
Bridging the representation gap with linear projection, Q-Former, perceiver resampler, MLP projectors, dynamic resolution, and architecture comparisons.
Contrastive learning foundations, InfoNCE loss, CLIP training pipeline, zero-shot transfer, ALIGN and Florence variants, and limitations.
Visual instruction tuning, GPT-4 data generation, two-stage training, LLaVA-1.5/NeXT, conversation formats, SFT data, and evaluation.
Pretraining objectives, web-scale data, supervised fine-tuning, RLHF for VLMs, DPO, hallucination mitigation, and training infrastructure.
LoRA, QLoRA, adapter tuning, prefix tuning, practical VLM fine-tuning strategies, and adapter merging for deployment.
Visual chain-of-thought, grounding, document understanding, video VLMs, multi-image reasoning, agentic VLMs, and spatial understanding.
VLM benchmarks, quantization, serving infrastructure, inference optimization, frontier models, open challenges, and future directions.
Curriculum designed to take you from vision-language fundamentals to production-ready multimodal systems.