ResNet
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
Read the Paper on arXivPaper Overview
ResNet (Residual Networks) introduced skip connections that allow training of much deeper networks than previously possible. Before ResNet, networks beyond ~20 layers suffered from the degradation problem -- deeper networks had higher training error, not due to overfitting, but due to optimization difficulties. He et al. demonstrated this empirically: a 56-layer plain network had higher training AND test error than its 20-layer counterpart on CIFAR-10, proving the problem was not capacity-related but optimization-related.
The key insight is deceptively simple: instead of asking a stack of layers to learn a desired mapping directly, let them learn the residual . The output becomes . If the optimal transformation is close to identity (which is common in deep networks where later layers refine rather than transform), pushing toward zero is far easier than learning an identity mapping through a stack of nonlinear layers. This is because the zero mapping is a natural attractor for weight decay and initialization near zero, whereas learning identity through conv-BN-ReLU stacks requires precise weight coordination.
ResNet won 1st place in the ILSVRC 2015 classification task with a 152-layer network achieving 3.57% top-5 error (surpassing human-level performance of ~5.1%). The same architecture also won 1st place in ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation -- a clean sweep across all five tracks. The 152-layer ResNet has 60M parameters and 11.3B FLOPs, yet trains stably with standard SGD and batch normalization.
Chapter Roadmap
Click any topic to jump in
Residual Learning: The Core Idea
Learn the residual $F(x) = H(x) - x$ instead of $H(x)$ directly — small perturbations from identity.
Gradient Flow and Vanishing Gradients
Skip connections turn a multiplicative gradient chain into an additive one — a highway that cannot vanish.
Bottleneck Architecture
1×1 → 3×3 → 1×1 squeeze-expand pattern — ~17× cheaper than basic blocks at matched width.
Network Depth and Performance
Plain nets degrade past ~20 layers; ResNets keep improving to 152+ — optimization was the bottleneck, not capacity.
Pre-Activation ResNet (Identity Mappings)
Moving ReLU/BN inside $F$ keeps the skip path a pure identity — enables 1000-layer training.
ResNeXt: Aggregated Residual Transformations
Aggregated transformations via grouped convs — cardinality as a new scaling axis beyond depth and width.
Squeeze-and-Excitation Networks (SE-Net)
Channel attention via GAP → MLP → sigmoid reweighting — a near-free accuracy boost.
Skip connections allow the network to learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. This reformulation does not add any parameters or computational complexity, yet fundamentally changes the optimization landscape.
The Problem
As networks get deeper, they become harder to train -- but not for the reason you might expect:
- A 56-layer plain network has higher training error than a 20-layer one on CIFAR-10. This is not overfitting -- the training loss itself is worse.
- This is the degradation problem: the optimization landscape becomes so complex that SGD cannot find a good solution, even though a solution provably exists (the 20-layer solution + identity layers).
- Batch normalization solved the vanishing/exploding gradient problem for ~20-30 layers, but degradation persists beyond that because the issue is not gradient magnitude but the difficulty of the loss surface.
- Mathematically, if you have a good 20-layer network and add 36 identity layers, the 56-layer network should be at least as good. But plain networks cannot learn these identity mappings through their nonlinear layer stacks -- the conv-BN-ReLU sequence makes identity a non-trivial function to represent.
- Experiments on ImageNet showed a 34-layer plain net had 28.54% top-1 error vs 25.74% for an 18-layer plain net -- adding layers made it worse.
The Solution
Residual Learning fundamentally changes what the network learns:
Instead of asking layers to learn directly, we reformulate: learn , then reconstruct .
Why this reformulation is so powerful:
-
Zero initialization advantage: If the optimal output is close to the input (common in deep networks where later layers make small refinements), only needs to be close to zero. With standard weight initialization (near zero) and weight decay pushing weights toward zero, the residual branch naturally starts near identity. A plain network must learn identity through conv-BN-ReLU, which is highly non-trivial.
-
Gradient highway: The addition means . The identity matrix provides an unconditional gradient path that does not decay with depth.
-
Loss surface smoothing: Li et al. (2018) later showed that skip connections make the loss landscape significantly smoother -- the loss surface of a ResNet has fewer local minima and saddle points compared to an equivalent plain network.
-
Ensemble interpretation: Veit et al. (2016) showed ResNets behave like ensembles of shallow networks. Removing any single residual block has minimal effect, suggesting the network learns a collection of paths of varying depth rather than a single deep pipeline.
Basic residual block (used in ResNet-18/34):
- Input: with channels at spatial size
- Conv 3x3 () BN ReLU Conv 3x3 () BN
- Add (element-wise), then ReLU
- Output: same shape
When spatial dimensions change (e.g., stride-2 downsampling from 56x56 to 28x28), a projection shortcut uses a 1x1 conv with stride 2 to match dimensions: where is a learned projection.
Key Points
Output: -- element-wise addition with zero extra parameters
Skip connection provides an identity mapping that costs nothing in compute or memory
When dimensions change (stride-2), a 1x1 projection shortcut matches spatial/channel dims
Residual blocks make identity the default -- the network only needs to learn deviations from identity
Empirically, learned residual responses have small magnitudes, confirming that layers learn small perturbations rather than complete transformations
ResNets behave as implicit ensembles of many shallow paths (Veit et al., 2016)
Mathematical Formulation
Residual Block Output
F(x) represents the residual mapping learned by stacked layers. When F has 2 layers: F = W₂ * ReLU(BN(W₁ * x)). The dimensions of F(x) and x must match for element-wise addition.
Projection Shortcut (dimension mismatch)
W_s is a 1x1 convolution that adjusts channels and/or spatial dimensions. Used when stride > 1 or input/output channels differ. Adds parameters but only at stage boundaries.
Instead of learning directly, a residual block learns , so the output is . If the optimal mapping is near-identity (common in very deep nets), only needs to learn a small perturbation — much easier than learning identity from scratch. The gradient becomes , with a guaranteed baseline.