Vision Language Models Study Plan

What VLMs are, the vision-language gap, evolution from VQA to GPT-4V, architecture overview, key capabilities, and the landscape of modern VLMs.

CNN feature extractors, Vision Transformers, CLIP visual encoder, SigLIP, InternViT, DINOv2, and choosing the right vision backbone.

Decoder-only vs encoder-decoder architectures, multimodal tokenization, attention mechanisms in VLMs, popular LLM backbones, and scaling laws.

Bridging the representation gap with linear projection, Q-Former, perceiver resampler, MLP projectors, dynamic resolution, and architecture comparisons.

Contrastive learning foundations, InfoNCE loss, CLIP training pipeline, zero-shot transfer, ALIGN and Florence variants, and limitations.

Visual instruction tuning, GPT-4 data generation, two-stage training, LLaVA-1.5/NeXT, conversation formats, SFT data, and evaluation.

Pretraining objectives, web-scale data, supervised fine-tuning, RLHF for VLMs, DPO, hallucination mitigation, and training infrastructure.

LoRA, QLoRA, adapter tuning, prefix tuning, practical VLM fine-tuning strategies, and adapter merging for deployment.

Visual chain-of-thought, grounding, document understanding, video VLMs, multi-image reasoning, agentic VLMs, and spatial understanding.

VLM benchmarks, quantization, serving infrastructure, inference optimization, frontier models, open challenges, and future directions.