Loading...
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
Read the Paper on arXivResNet (Residual Networks) introduced skip connections that allow training of much deeper networks than previously possible. Before ResNet, networks beyond ~20 layers suffered from the degradation problem -- deeper networks had higher training error, not due to overfitting, but due to optimization difficulties. He et al. demonstrated this empirically: a 56-layer plain network had higher training AND test error than its 20-layer counterpart on CIFAR-10, proving the problem was not capacity-related but optimization-related.
The key insight is deceptively simple: instead of asking a stack of layers to learn a desired mapping directly, let them learn the residual . The output becomes . If the optimal transformation is close to identity (which is common in deep networks where later layers refine rather than transform), pushing toward zero is far easier than learning an identity mapping through a stack of nonlinear layers. This is because the zero mapping is a natural attractor for weight decay and initialization near zero, whereas learning identity through conv-BN-ReLU stacks requires precise weight coordination.
ResNet won 1st place in the ILSVRC 2015 classification task with a 152-layer network achieving 3.57% top-5 error (surpassing human-level performance of ~5.1%). The same architecture also won 1st place in ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation -- a clean sweep across all five tracks. The 152-layer ResNet has 60M parameters and 11.3B FLOPs, yet trains stably with standard SGD and batch normalization.
Click any topic to jump in
Learn the residual $F(x) = H(x) - x$ instead of $H(x)$ directly — small perturbations from identity.
Skip connections turn a multiplicative gradient chain into an additive one — a highway that cannot vanish.
1×1 → 3×3 → 1×1 squeeze-expand pattern — ~17× cheaper than basic blocks at matched width.
Plain nets degrade past ~20 layers; ResNets keep improving to 152+ — optimization was the bottleneck, not capacity.
Moving ReLU/BN inside $F$ keeps the skip path a pure identity — enables 1000-layer training.
Aggregated transformations via grouped convs — cardinality as a new scaling axis beyond depth and width.
Channel attention via GAP → MLP → sigmoid reweighting — a near-free accuracy boost.
Skip connections allow the network to learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. This reformulation does not add any parameters or computational complexity, yet fundamentally changes the optimization landscape.
As networks get deeper, they become harder to train -- but not for the reason you might expect:
Residual Learning fundamentally changes what the network learns:
Instead of asking layers to learn directly, we reformulate: learn , then reconstruct .
Why this reformulation is so powerful:
Zero initialization advantage: If the optimal output is close to the input (common in deep networks where later layers make small refinements), only needs to be close to zero. With standard weight initialization (near zero) and weight decay pushing weights toward zero, the residual branch naturally starts near identity. A plain network must learn identity through conv-BN-ReLU, which is highly non-trivial.
Gradient highway: The addition means . The identity matrix provides an unconditional gradient path that does not decay with depth.
Loss surface smoothing: Li et al. (2018) later showed that skip connections make the loss landscape significantly smoother -- the loss surface of a ResNet has fewer local minima and saddle points compared to an equivalent plain network.
Ensemble interpretation: Veit et al. (2016) showed ResNets behave like ensembles of shallow networks. Removing any single residual block has minimal effect, suggesting the network learns a collection of paths of varying depth rather than a single deep pipeline.
Basic residual block (used in ResNet-18/34):
When spatial dimensions change (e.g., stride-2 downsampling from 56x56 to 28x28), a projection shortcut uses a 1x1 conv with stride 2 to match dimensions: where is a learned projection.
Output: -- element-wise addition with zero extra parameters
Skip connection provides an identity mapping that costs nothing in compute or memory
When dimensions change (stride-2), a 1x1 projection shortcut matches spatial/channel dims
Residual blocks make identity the default -- the network only needs to learn deviations from identity
Empirically, learned residual responses have small magnitudes, confirming that layers learn small perturbations rather than complete transformations
ResNets behave as implicit ensembles of many shallow paths (Veit et al., 2016)
F(x) represents the residual mapping learned by stacked layers. When F has 2 layers: F = W₂ * ReLU(BN(W₁ * x)). The dimensions of F(x) and x must match for element-wise addition.
W_s is a 1x1 convolution that adjusts channels and/or spatial dimensions. Used when stride > 1 or input/output channels differ. Adds parameters but only at stage boundaries.
Instead of learning directly, a residual block learns , so the output is . If the optimal mapping is near-identity (common in very deep nets), only needs to learn a small perturbation — much easier than learning identity from scratch. The gradient becomes , with a guaranteed baseline.