Instance Segmentation2017

Mask R-CNN

Instance Segmentation Framework

Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick

Paper Overview

Mask R-CNN extends Faster R-CNN by adding a parallel branch for predicting per-instance segmentation masks, achieving state-of-the-art instance segmentation while maintaining the speed of the two-stage detection framework.

The architecture follows a clear data flow: an input image (e.g., 800x1024) passes through a ResNet-50-FPN or ResNet-101-FPN backbone producing multi-scale feature maps (P2 through P5 at strides 4, 8, 16, 32 respectively, each with 256 channels). A Region Proposal Network generates ~1000 proposals per FPN level, which are filtered to ~300 via NMS. Each proposal is then processed by RoIAlign to extract fixed-size features (7x7x256 for classification/box heads, 14x14x256 for the mask head). Three parallel heads produce the final outputs: class probabilities over K+1 categories, 4 bounding box regression offsets, and K binary masks at 28x28 resolution.

The key insight is that adding a small fully convolutional network (FCN) that predicts a binary mask for each Region of Interest (RoI), running in parallel with the existing classification and bounding box heads, requires minimal additional computation (~20% overhead) while enabling pixel-precise segmentation. Critically, the mask head predicts K independent binary masks rather than a single K-way softmax mask, decoupling mask prediction from classification and eliminating inter-class competition.

Key advances:

RoIAlign: Replaces RoI Pooling's integer quantization with bilinear interpolation at exact floating-point coordinates, fixing spatial misalignment that costs ~3 AP in mask quality
Parallel mask branch: Per-class binary masks decoupled from classification -- K independent masks, not K-way softmax. This single design choice accounts for +3.6 AP improvement
Feature Pyramid Network (FPN): Top-down pathway with lateral connections creates 256-channel feature maps at 4 resolutions (strides 4/8/16/32), enabling detection of objects from ~20px to ~800px
Multi-task training: Joint loss ( $\mathcal{L}_{cls} + \mathcal{L}_{box} + \mathcal{L}_{mask}$ ) with equal weighting improves all tasks vs. training separately -- the mask task regularizes the shared backbone
Generalizable framework: Same architecture handles instance segmentation, keypoint detection (K=17 for COCO human pose), panoptic segmentation, and DensePose by swapping only the prediction head

COCO benchmark results (ResNet-101-FPN backbone):

37.1 AP on instance segmentation (mask), 39.8 AP on object detection (box)
62.7 AP on keypoint detection using the same framework with a keypoint head
Runs at 5 FPS on a single GPU (Nvidia Tesla M40), with the mask branch adding only 20% overhead over Faster R-CNN
Total model parameters: ~44M (ResNet-50-FPN) or ~63M (ResNet-101-FPN)

Mask R-CNN established the "detect then segment" paradigm that became the foundation for Detectron2 and countless downstream applications in medical imaging, autonomous driving, robotics, and video analysis.

Chapter Roadmap

Click any topic to jump in

Two-Stage Pipeline

RPN proposes regions then a head classifies and refines them.

feeds

FPN

Multi-scale feature pyramid for detecting objects across sizes.

RoIAlign

Bilinear-interpolation pooling that preserves pixel alignment.

Mask Branch

Per-class FCN producing 28x28 masks per RoI.

enables

Multi-Task Loss

Joint classification, box regression, and mask losses.

Beyond Masks

Same head template extends to keypoints, person parsing, and more.

Mask R-CNN inherits Faster R-CNN's two-stage architecture: first propose candidate regions with a Region Proposal Network, then classify and refine each proposal independently through three parallel heads. Understanding this pipeline end-to-end -- from raw pixels to instance masks -- is essential to grasping why each component exists and how they interact.

The Problem

The detection challenge:

Object detection requires locating every object in an image and assigning a class label. Two competing paradigms existed before Mask R-CNN:

Single-stage detectors (YOLO, SSD):

Directly predict class and bounding box from a dense grid of anchors in a single forward pass
Fast (real-time, 30-60 FPS) but less accurate, especially for small objects and dense scenes
No explicit "proposal" step -- must handle the extreme imbalance of background vs. object (99%+ of anchors are background) in a single classification pass
SSD uses features from multiple layers (conv4_3, conv7, etc.) but early layers lack semantic richness, leading to poor small-object detection (AP for small objects ~10-15% lower than two-stage methods)
Struggle with extreme aspect ratios, heavily occluded objects, and scenes with 50+ instances

Why two stages fundamentally help:

The two-stage decomposition splits an intractable problem into two tractable sub-problems:

Stage 1 (RPN) focuses on objectness -- a simple binary task (object vs. background). Because it only needs to answer "is something here?", it can use lightweight 3x3 conv sliding windows with ~10 anchor templates per location. This is computationally cheap and achieves >95% recall at 300 proposals.

Stage 2 focuses on fine-grained understanding -- per-proposal classification across K classes, precise bounding box refinement, and now mask prediction. Because it only processes ~300 proposals (not ~100,000 anchors), it can afford:

Larger feature maps per proposal (7x7 or 14x14 via RoIAlign vs. single grid cells)
More computation per proposal (2 FC layers with 1024 units each for class/box)
Higher spatial resolution for mask prediction (14x14 input, 28x28 output)

Quantitative advantage: On COCO, two-stage Faster R-CNN achieves 36.2 box AP vs. single-stage SSD's 28.8 AP (same ResNet-101 backbone). The gap is largest on small objects: 18.2 AP vs. 10.1 AP.

The missing piece:

Faster R-CNN produces class labels and bounding boxes per object but has no pixel-level understanding. It knows a "dog" is at coordinates (120, 85, 350, 290) but not which pixels within that box are dog vs. background. Adding instance segmentation requires predicting a binary mask for each detected object -- and this mask must be spatially precise to sub-pixel accuracy, which is where RoI Pooling's quantization becomes a critical bottleneck.

The Solution

Mask R-CNN's complete two-stage pipeline with concrete dimensions:

Consider an input image of size 800x1024 (height x width) processed through the full pipeline:

Stage 1: Region Proposal Network (RPN)

The RPN operates on FPN feature maps (P2 through P5). At each spatial position on each level, it evaluates $k = 3$ anchor aspect ratios (1:2, 1:1, 2:1) at a single scale per level (P2: 32px, P3: 64px, P4: 128px, P5: 256px, P6: 512px). This generates:

P2 (200x256 spatial): 200 x 256 x 3 = 153,600 anchors
P3 (100x128): 100 x 128 x 3 = 38,400 anchors
P4 (50x64): 50 x 64 x 3 = 9,600 anchors
P5 (25x32): 25 x 32 x 3 = 2,400 anchors
P6 (13x16): 13 x 16 x 3 = 624 anchors
Total: ~204,000 anchors across all levels

For each anchor, a lightweight head (single 3x3 conv with 256 channels, then two sibling 1x1 convs) predicts:

Objectness score: $p_{obj} \in [0, 1]$ -- probability that the anchor contains any object
Box regression: 4 offsets $(\Delta x, \Delta y, \Delta w, \Delta h)$ to refine the anchor into a tighter box

$p_{\text{obj}}, \Delta x, \Delta y, \Delta w, \Delta h = \text{RPN}(\text{features at anchor location})$

NMS filtering: The top 2000 anchors by objectness score (per FPN level) undergo Non-Maximum Suppression with IoU threshold 0.7, yielding ~1000 proposals total. During training, 256 proposals are sampled (1:1 positive/negative ratio) for RPN loss. During inference, the top 300 proposals (by post-NMS score) proceed to Stage 2.

RPN parameter count: Only ~2.4M parameters (one shared 3x3 conv + two 1x1 heads), since it shares the backbone with Stage 2.

Stage 2: Three parallel heads per proposal

Each surviving proposal is mapped to the appropriate FPN level via the scale assignment formula, then processed by RoIAlign to extract fixed-size features:

For box/class heads: RoIAlign extracts 7x7x256 features per proposal
For mask head: RoIAlign extracts 14x14x256 features per proposal (larger for spatial precision)

The three parallel prediction heads:

Classification head (FC-based): 7x7x256 = 12,544 → flatten → FC(1024) → ReLU → FC(1024) → ReLU → FC(K+1) → softmax. Outputs K+1 class probabilities (K=80 for COCO + 1 background). Parameters: ~26M
Box regression head (FC-based): Same architecture → FC(4K). Outputs 4 refined coordinates per class (class-specific regression). Parameters: ~26M
Mask head (FCN-based, new in Mask R-CNN): 14x14x256 → 4 consecutive 3x3 conv layers (each 256 channels, with ReLU) → 2x2 transposed conv (stride 2) → 28x28x256 → 1x1 conv → 28x28xK binary masks. Parameters: ~2.4M

The mask head adds only ~20% computational overhead to Faster R-CNN because it operates on just ~300 proposals and uses lightweight convolutions rather than FC layers.

Full inference pipeline (end-to-end):

Run backbone (ResNet-50/101) + FPN → P2, P3, P4, P5 feature maps (each 256 channels)
Run RPN on all FPN levels → ~204K anchors → NMS → 300 proposals
Assign each proposal to its FPN level via scale formula
RoIAlign → 7x7x256 (box/class) and 14x14x256 (mask) features per proposal
Run all three heads in parallel on extracted features
Apply class-specific NMS (IoU 0.5) to box predictions → ~100 detections
For each surviving detection with class $k$ , select the $k$ -th mask from the 28x28xK output
Bilinearly upsample the 28x28 mask to the detection bounding box size

Inference speed: ~200ms per image (5 FPS) on a single Nvidia Tesla M40 GPU, with backbone taking ~120ms, RPN ~20ms, RoIAlign + heads ~50ms, and post-processing ~10ms.

Key Points

Stage 1 (RPN): 3x3 conv + two 1x1 heads evaluate ~204K anchors across 5 FPN levels, NMS reduces to ~300 proposals with >95% recall

Stage 2: Three parallel heads on RoIAlign features -- classification (7x7 -> FC -> K+1 classes), box regression (7x7 -> FC -> 4K coords), mask prediction (14x14 -> FCN -> 28x28xK masks)

Mask head adds only ~2.4M parameters and ~20% compute overhead to the ~44M parameter Faster R-CNN baseline

Two-stage achieves +7.4 box AP over single-stage SSD on COCO (36.2 vs 28.8), with the gap largest on small objects (+8.1 AP)

RPN shares the entire backbone with Stage 2 -- proposals cost only ~20ms of the ~200ms total inference time

End-to-end pipeline: 800x1024 image -> backbone (120ms) -> RPN (20ms) -> RoIAlign + heads (50ms) -> post-processing (10ms) -> instance masks at 5 FPS

Mathematical Formulation

RPN Output (per anchor)

$p_{\text{obj}}, (\Delta x, \Delta y, \Delta w, \Delta h) = \text{RPN}(F_{\text{backbone}})$

Each of the ~204K anchors produces an objectness probability and 4 box refinement offsets. The RPN head is a single 3x3 conv (shared across all anchors at a level) followed by two sibling 1x1 convs for classification and regression. Top-scoring anchors after NMS (IoU threshold 0.7) become the ~300 proposals for Stage 2.

Mask R-CNN Output (per proposal)

$\text{class}, \text{box}, \text{mask}_{28 \times 28} = \text{Heads}(\text{RoIAlign}(F, \text{proposal}))$

Each proposal is processed by three parallel heads. The classification head produces K+1 softmax probabilities. The box head produces 4K regression offsets (class-specific). The mask head produces K independent 28x28 binary masks. At inference, the predicted class k selects which mask to output and upsample to the bounding box size.

Anchor Generation (per FPN level)

$N_{\text{anchors}} = H_l \times W_l \times |\text{ratios}|$

Each FPN level l generates anchors at every spatial position with 3 aspect ratios (1:2, 1:1, 2:1) at scale 2^(l+1) pixels. For an 800x1024 input: P2 generates 153,600 anchors (32px base), P3 generates 38,400 (64px), P4 generates 9,600 (128px), P5 generates 2,400 (256px), totaling ~204K anchors.

Mathematical Intuition

Stage 1 (RPN) generates ~ $2000$ proposals from anchors, scored via objectness $p_{\text{obj}}$ and refined via box regression $\Delta = (\Delta_x, \Delta_y, \Delta_w, \Delta_h)$ . Stage 2 classifies and refines each proposal. Total cost: $O(HW \cdot A)$ for RPN (anchors) plus $O(N_{\text{prop}} \cdot k^2 \cdot C)$ for the head — much cheaper than dense one-stage prediction at high accuracy.

Architecture

Complete data flow through the two-stage pipeline:

Input: Image tensor [1, 3, 800, 1024]

Backbone (ResNet-50):

conv1 + pool: [1, 64, 200, 256]
C2 (res2): [1, 256, 200, 256] -- stride 4
C3 (res3): [1, 512, 100, 128] -- stride 8
C4 (res4): [1, 1024, 50, 64] -- stride 16
C5 (res5): [1, 2048, 25, 32] -- stride 32

FPN (top-down with lateral connections):

P5 = Conv1x1(C5): [1, 256, 25, 32]
P4 = Upsample(P5) + Conv1x1(C4): [1, 256, 50, 64]
P3 = Upsample(P4) + Conv1x1(C3): [1, 256, 100, 128]
P2 = Upsample(P3) + Conv1x1(C2): [1, 256, 200, 256]
Each level refined by 3x3 conv to reduce aliasing

RPN (per FPN level):

Shared 3x3 conv: [1, 256, H, W] per level
Classification branch (1x1 conv): [1, 3, H, W] -- 3 objectness scores per position
Regression branch (1x1 conv): [1, 12, H, W] -- 4 offsets x 3 anchors per position
NMS (IoU 0.7): ~204K anchors -> ~1000 proposals -> top 300

Stage 2 (per proposal):

RoIAlign (box/class): [300, 256, 7, 7]
RoIAlign (mask): [300, 256, 14, 14]
Classification: [300, 256, 7, 7] -> flatten -> FC(1024) -> FC(1024) -> FC(81) -> [300, 81]
Box regression: [300, 256, 7, 7] -> flatten -> FC(1024) -> FC(1024) -> FC(324) -> [300, 81, 4]
Mask: [300, 256, 14, 14] -> 4x Conv3x3(256) -> Deconv2x2 -> Conv1x1(80) -> [300, 80, 28, 28]

Premium Content

Upgrade to PixelBank Premium to unlock this content.

Back to All Papers

Study Plans

Computer Vision Study Plan All Papers Practice Problems

Mask R-CNN

Paper Overview

Chapter Roadmap

Two-Stage Pipeline

FPN

RoIAlign

Mask Branch

Multi-Task Loss

Beyond Masks

Two-Stage Detection Pipeline

The Problem

The Solution

Key Points

Mathematical Formulation

RPN Output (per anchor)

Mask R-CNN Output (per proposal)

Anchor Generation (per FPN level)

Architecture

RoIAlign

Mask Prediction Branch

Feature Pyramid Network (FPN)

Multi-Task Training

Beyond Instance Segmentation

Premium Content

Related Papers

Study Plans