Segment Anything Model 3 (SAM 3)

SAM 3 (Segment Anything Model 3) is Meta's latest foundation model for detection, segmentation, and tracking of objects in images and videos using concept prompts — text descriptions, visual exemplars, or both. It represents a fundamental leap from SAM 1's click-based prompting and SAM 2's video memory architecture.

Architectural lineage and what changed:

SAM 1 (2023) introduced the "segment anything" paradigm with a three-component architecture: a ViT-H image encoder (632M params, processing 1024x1024 inputs into 64x64x256 embeddings), a prompt encoder handling points/boxes/masks/text, and a lightweight 2-layer Transformer mask decoder producing 3 ambiguity-aware masks with confidence scores. It was trained on SA-1B (11M images, 1.1B masks) using a three-phase data engine (assisted, semi-automatic, fully automatic annotation). SAM 2 (2024) extended this with streaming memory for video — a memory encoder, memory bank, and memory attention module enabling temporal propagation across frames.

SAM 3 (2025) introduces a fundamentally new ability: open-vocabulary concept segmentation. Rather than clicking individual objects, you describe a concept ("person in red shirt") and SAM 3 finds and segments every instance across images or videos. This required rethinking the entire architecture:

Key advances:

• Concept prompts: Text descriptions, visual exemplars, or combined prompts — replacing point/box clicking with semantic queries
• Perception Encoder (PE): Joint vision-language backbone pre-trained on 5.4B image-text pairs using contrastive learning, replacing SAM 1's vision-only ViT-H encoder
• Presence token: A learned query token that discriminates between closely related prompts (e.g., "player in white" vs "player in red") via binary classification with presence probability
• Decoupled detector-tracker: Separates object detection (runs once per keyframe) from temporal tracking (propagates per frame), eliminating the task interference of joint models
• SA-Co dataset: 4 million unique concepts annotated via an automated CLIP+LLM pipeline built on SA-1B — 3,300x more concept vocabulary than LVIS (1,200 categories)

SAM 3 achieves 2x performance gain over existing systems on the Promptable Concept Segmentation benchmark while maintaining SAM 2's real-time video tracking capabilities. The decoupled architecture scales as O(objects) in memory rather than O(frames x objects), enabling efficient processing of long videos with many tracked instances.