ResNet

In deep networks without skip connections, gradients during backpropagation face compounding attenuation:

• At each layer $l$, the gradient is multiplied by the local Jacobian $\frac{\partial h_l}{\partial h_{l-1}}$, which depends on the weight matrix and activation function derivative.
• For a network of depth $L$, the gradient at layer 1 involves a product of $L$ Jacobian matrices: $\frac{\partial L}{\partial h_1} = \prod_{l=2}^{L} \frac{\partial h_l}{\partial h_{l-1}}$.
• If each Jacobian has spectral norm < 1 (common with sigmoid/tanh activations), gradients decay exponentially: after 50 layers, a factor of $0.9^{50} \approx 0.005$ -- effectively zero.
• ReLU helped by having gradient exactly 1 for positive inputs, but dead ReLU units (gradient = 0) create "holes" in the gradient path.
• Batch normalization stabilized gradient magnitudes within individual layers but could not prevent the multiplicative decay across many layers.
• The practical consequence: in a 56-layer plain network, the first 20 layers receive gradients so small that their learning rate is effectively zero -- they stop contributing to the network's representation.

Skip connections create gradient highways with a guaranteed minimum gradient magnitude:

For a single residual block with output $y = F(x) + x$, the backpropagated gradient is:

$\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \left(1 + \frac{\partial F}{\partial x}\right)$

The +1 term is the critical insight. It means:

1. Even if the residual branch gradient $\frac{\partial F}{\partial x}$ vanishes completely (equals 0), the gradient through the skip path is exactly the upstream gradient -- no attenuation at all.

2. For a chain of $L$ residual blocks, the gradient from loss $\mathcal{L}$ to the input $x_0$ can be decomposed as a sum over all possible paths through the network. One of those paths is the "direct path" through all skip connections, which has gradient magnitude 1 regardless of depth.

Formally, for a chain of residual blocks:

$x_L = x_0 + \sum_{i=0}^{L-1} F_i(x_i)$

$\frac{\partial \mathcal{L}}{\partial x_0} = \frac{\partial \mathcal{L}}{\partial x_L} \cdot \left(1 + \frac{\partial}{\partial x_0} \sum_{i=0}^{L-1} F_i(x_i)\right)$

The "1" term ensures the gradient signal from the loss reaches every layer without multiplicative decay. The residual terms $\frac{\partial F_i}{\partial x_0}$ provide additional gradient signal but are not required for training to work.

Practical impact: He et al. showed that a 1202-layer ResNet trains successfully on CIFAR-10 (though it slightly overfits due to the small dataset). A 1202-layer plain network would be completely untrainable.

Scaling basic residual blocks (two 3x3 convs) to 50+ layers creates a computational wall:

• At the deepest stage (conv5_x), feature maps have 512 channels at 7x7 spatial resolution.
• A single basic block at this stage: two 3x3 convolutions with 512 input and 512 output channels.
• Parameters per block: $2 \times (512 \times 512 \times 3 \times 3) = 4.72M$ parameters.
• With 3 blocks at this stage alone: 14.16M parameters just for the final stage.
• FLOPs per block at 7x7: $2 \times (512 \times 512 \times 3 \times 3 \times 7 \times 7) = 231M$ FLOPs.
• For a 50-layer network with basic blocks, total parameters would exceed 80M and FLOPs would be impractical.
• The 3x3 convolution is the bottleneck: it scales as $O(C^2)$ in parameters, making wide networks expensive.

The bottleneck block uses a 3-layer design that squeezes channels through a narrow "bottleneck" before expanding back:

Layer-by-layer data flow (for the conv4_x stage with 1024-channel input):

1. 1x1 conv (reduce): $1024 \to 256$ channels. Parameters: $1024 \times 256 = 262K$. This is a learned dimensionality reduction -- it projects the high-dimensional feature into a compact representation.

2. 3x3 conv (transform): $256 \to 256$ channels. Parameters: $256 \times 256 \times 3 \times 3 = 590K$. The expensive spatial convolution now operates on only 256 channels instead of 1024, reducing cost by $16\times$.

3. 1x1 conv (expand): $256 \to 1024$ channels. Parameters: $256 \times 1024 = 262K$. Restores the original channel dimension so the skip connection addition works.

Total parameters per bottleneck block: ~1.11M vs ~18.87M for equivalent basic blocks ($1024 \times 1024 \times 9 \times 2$). That is a ~17x reduction.

The expansion ratio $r = 4$ is used throughout: the bottleneck width is always $C/4$ where $C$ is the output channels. At each stage:

Stage	Output Channels	Bottleneck Width	Spatial Size
conv2_x	256	64	56x56
conv3_x	512	128	28x28
conv4_x	1024	256	14x14
conv5_x	2048	512	7x7

Each BN layer follows each conv layer, and ReLU follows each BN (in the original post-activation design). The skip connection adds the input directly to the output of the third BN layer, followed by a final ReLU.

Why 1x1 convolutions work: Despite having no spatial extent, 1x1 convolutions perform cross-channel feature recombination. They act as learned linear projections across the channel dimension, enabling the network to discover which channel combinations are informative before the expensive 3x3 spatial processing.

Before ResNet, the relationship between depth and performance was broken:

• VGG-16/19 (2014) achieved 7.3% top-5 error on ImageNet with 138M parameters, but attempts to go deeper than 19 layers consistently failed.
• A 34-layer plain network scored 28.54% top-1 error vs 25.74% for an 18-layer version -- 3.8% degradation from adding layers.
• GoogLeNet/Inception used 22 layers but relied on complex multi-branch topology rather than simple stacking.
• No principled way existed to scale depth: each architecture was hand-designed and tuning deeper versions required extensive engineering.
• The fundamental question was unresolved: is there a depth beyond which adding layers always hurts, or is it purely an optimization failure?

ResNet answered definitively: depth degradation is an optimization problem, not a capacity problem, and skip connections solve it. The family of ResNet models demonstrates smooth, predictable scaling:

Complete ResNet architecture specifications:

Model	Layers	Block Type	Stage Config [conv2-5]	Params	FLOPs	Top-1	Top-5
ResNet-18	18	Basic	[2, 2, 2, 2]	11.7M	1.8B	69.8%	89.1%
ResNet-34	34	Basic	[3, 4, 6, 3]	21.8M	3.6B	73.3%	91.4%
ResNet-50	50	Bottleneck	[3, 4, 6, 3]	25.6M	3.8B	76.1%	93.0%
ResNet-101	101	Bottleneck	[3, 4, 23, 3]	44.5M	7.6B	77.4%	93.5%
ResNet-152	152	Bottleneck	[3, 8, 36, 3]	60.2M	11.3B	78.3%	94.2%

Shared architecture skeleton (all variants):

1. conv1: 7x7 conv, 64 filters, stride 2 $\to$ output: 112x112x64
2. pool: 3x3 max pool, stride 2 $\to$ output: 56x56x64
3. conv2_x: Blocks at 56x56 with 64/256 channels
4. conv3_x: Blocks at 28x28 with 128/512 channels (first block stride 2)
5. conv4_x: Blocks at 14x14 with 256/1024 channels (first block stride 2)
6. conv5_x: Blocks at 7x7 with 512/2048 channels (first block stride 2)
7. Global average pool: 7x7 $\to$ 1x1
8. FC layer: 2048 (or 512 for basic) $\to$ 1000 classes

Key observations from the results:

• ResNet-34 vs ResNet-18: +3.5% top-1 with 2x depth -- residual learning makes this gain reliable.
• ResNet-50 vs ResNet-34: switching to bottleneck blocks gains +2.8% while only adding 3.8M parameters (17% more) because bottleneck blocks are more parameter-efficient.
• ResNet-152: 3.57% top-5 error won ILSVRC 2015, beating all ensemble methods from 2014. A single model outperformed the previous year's ensemble winner.
• An ensemble of 6 ResNet models achieved 3.08% top-5 error.
• Diminishing returns appear beyond 152 layers on ImageNet (1.2M images), but the ceiling is dataset-dependent, not architecture-dependent.

The original (post-activation) ResNet block has this structure:

x → Conv → BN → ReLU → Conv → BN → (+x) → ReLU → y

While this works for 152 layers, several issues emerge at extreme depths:

• The final ReLU after addition clips the output to non-negative values, meaning the skip connection is not a true identity: $y = \text{ReLU}(F(x) + x) \neq F(x) + x$. This non-linearity on the shortcut path breaks the clean gradient highway.
• BN after conv means the skip connection signal passes through BN and ReLU before reaching subsequent blocks, accumulating non-linear distortions.
• At 1000+ layers, these distortions compound: the information signal through the skip path degrades, and the gradient highway develops "toll booths" at every ReLU gate.
• He et al. tested a 1001-layer post-activation ResNet on CIFAR-10 and found it trained but showed signs of optimization difficulty compared to pre-activation.
• The paper systematically tested 5 different shortcut and activation arrangements to find the optimal design.

Pre-activation rearranges the block so that BN and ReLU are applied to the input before the weight layers:

x → BN → ReLU → Conv → BN → ReLU → Conv → (+x) → y

Why this is better -- three key insights:

1. Pure identity shortcut: The skip connection is now $y = x + F(\hat{x})$ where $\hat{x} = \text{ReLU}(\text{BN}(x))$ is the pre-activated input. The addition is the LAST operation, with no ReLU gate after it. This means the skip path is a true identity mapping: information and gradients flow through without any non-linear distortion.

2. Clean gradient propagation: For a stack of $L$ pre-activation blocks: $x_L = x_0 + \sum_{i=0}^{L-1} F_i(\hat{x}_i)$ $\frac{\partial \mathcal{L}}{\partial x_0} = \frac{\partial \mathcal{L}}{\partial x_L} \left(1 + \frac{\partial}{\partial x_0}\sum_{i=0}^{L-1} F_i(\hat{x}_i)\right)$

The gradient from loss to ANY layer always contains the additive "1" term -- no multiplicative gates anywhere on the identity path.

3. BN as pre-conditioning: Moving BN before the conv means each conv layer receives normalized input. This is analogous to "pre-layer normalization" in Transformers (which was discovered later and provides similar benefits for deep Transformer training).

Empirical results on CIFAR-10:

• 164-layer pre-activation: 5.46% error (vs 5.93% post-activation)
• 1001-layer pre-activation: 4.92% error (trains cleanly!)
• 1001-layer post-activation: 7.61% error (optimization struggles)

The improvement grows with depth, confirming the theoretical advantage of pure identity shortcuts.

After ResNet established depth scaling, the question became: how to increase capacity without diminishing returns?

Three conventional scaling dimensions and their limitations:

1. Depth (more layers): Returns diminish -- ResNet-200 barely improves over ResNet-152, and training time grows linearly.
2. Width (more channels): FLOPs and parameters grow quadratically with channel count. Doubling width quadruples the cost of 3x3 convolutions.
3. Kernel size: Larger kernels are expensive ($k^2$ cost) and VGG showed 3x3 stacks outperform single large kernels.

The Inception insight, but simpler: GoogLeNet/Inception showed that multi-branch topology helps, but Inception modules require careful per-layer design with different kernel sizes (1x1, 3x3, 5x5). Can we get the benefits of multi-branch processing without the architectural complexity?

ResNeXt introduces cardinality $C$ -- the number of independent transformation paths within each block. All paths have identical topology (unlike Inception), making the design simple and modular.

How a ResNeXt bottleneck block works (ResNeXt-50 32x4d, conv4_x stage):

Input: $14 \times 14 \times 1024$

Option A (equivalent grouped convolution implementation):

1. 1x1 conv: $1024 \to 128$ channels (32 groups $\times$ 4 channels each)
2. 3x3 grouped conv: $128 \to 128$ with $C=32$ groups (each group: $4 \to 4$ channels)
3. 1x1 conv: $128 \to 1024$ channels

Option B (conceptual split-transform-merge):

• Split input into 32 paths via 32 separate 1x1 convs ($1024 \to 4$ each)
• Each path: 1x1 ($4$) $\to$ 3x3 ($4$) $\to$ 1x1 ($1024/32 = 32$? No -- all paths share the output 1x1)
• Actually: each of 32 paths does $1024 \to 4 \to 4 \to 4$, then concatenate to 128, then 1x1 to 1024

These are mathematically equivalent but grouped convolution is the efficient implementation.

Parameter comparison with ResNet-50:

• ResNet-50 bottleneck (conv4_x): $1024 \times 256 + 256^2 \times 9 + 256 \times 1024 = 1.11M$ per block
• ResNeXt-50 32x4d (conv4_x): $1024 \times 128 + 128 \times 128/32 \times 9 \times 32 + 128 \times 1024 = 1.05M$ per block (similar params, but distributed differently)

Why cardinality helps: Each group learns a different transformation subspace. The aggregation of 32 diverse 4-channel transformations captures richer representations than a single 128-channel transformation, even with the same total parameters. This is related to ensemble learning -- diversity among weak learners often outperforms a single strong learner.

Results on ImageNet:

• ResNeXt-50 32x4d: 77.8% top-1 (vs ResNet-50: 76.1%, ResNet-101: 77.4%)
• ResNeXt-101 32x4d: 78.8% top-1
• ResNeXt-101 64x4d: 79.6% top-1 (at higher compute)

Standard convolutions treat all output channels identically -- each filter produces one channel, and all channels are passed to the next layer with equal importance:

• A 3x3 conv with 256 output channels produces 256 feature maps, but some are far more informative than others for the current input.
• A channel detecting "fur texture" is critical for classifying a cat but irrelevant for classifying a car. Yet both channels contribute equally to subsequent layers.
• Without explicit channel selection, the network must use deeper layers to implicitly learn to ignore irrelevant channels, wasting capacity and depth.
• Global context is needed to decide channel importance (e.g., knowing the overall scene determines which features matter), but standard convolutions operate locally.
• Attention mechanisms existed for spatial dimensions (e.g., in detection), but channel-wise attention was unexplored for classification.

The Squeeze-and-Excitation (SE) block adds a lightweight channel attention mechanism in three steps:

Step 1: Squeeze (Global Average Pooling) Collapse each channel's $H \times W$ spatial map into a single scalar by global average pooling: $z_c = \frac{1}{H \times W} \sum_{i=1}^{H} \sum_{j=1}^{W} x_c(i, j)$

This produces a channel descriptor $z \in \mathbb{R}^C$ that captures the global distribution of each channel. For a $14 \times 14 \times 1024$ feature map, squeeze produces a $1 \times 1 \times 1024$ vector.

Step 2: Excitation (Bottleneck FC Network) A two-layer fully connected network learns non-linear channel interdependencies: $s = \sigma(W_2 \cdot \delta(W_1 \cdot z))$

Where:

• $W_1 \in \mathbb{R}^{C/r \times C}$ reduces channels by ratio $r$ (typically $r=16$): $1024 \to 64$
• $\delta$ is ReLU activation
• $W_2 \in \mathbb{R}^{C \times C/r}$ expands back: $64 \to 1024$
• $\sigma$ is sigmoid, outputting values in $[0, 1]$

Parameters for this FC bottleneck: $2 \times C \times C/r = 2 \times 1024 \times 64 = 131K$ per block.

Step 3: Scale (Channel-wise Multiplication) Multiply each channel by its learned attention weight: $\tilde{x}_c = s_c \cdot x_c$

Channels with $s_c \approx 1$ are amplified (important), channels with $s_c \approx 0$ are suppressed (irrelevant).

Integration with ResNet: The SE block is inserted within each residual block, after the conv layers but before the skip connection addition:

x → [Conv layers → SE block] + x → ReLU → y

SE-ResNet-50 overhead:

• Extra parameters: ~2.5M (10% of 25.6M base)
• Extra FLOPs: negligible (FC on 1x1 vectors)
• Top-1 improvement: 76.1% $\to$ 77.6% (+1.5%)

ImageNet 2017 result: SE-ResNeXt-101 achieved 2.251% top-5 error, winning the last ImageNet classification competition.