Training2016

Normalization Techniques

BatchNorm, LayerNorm, GroupNorm & InstanceNorm

Sergey Ioffe, Jimmy Lei Ba, Yuxin Wu, Dmitry Ulyanov

Paper Overview

Normalization layers are essential building blocks in modern deep learning. They stabilize training, enable higher learning rates, and reduce sensitivity to initialization — but different normalization types operate over different dimensions, making them suited for different architectures.

Batch Normalization (2015) normalizes across the batch dimension, making it ideal for CNNs with large batches. Layer Normalization (2016) normalizes across features per sample, becoming the standard for Transformers. Group Normalization (2018) and Instance Normalization (2016) offer alternatives for small-batch and style transfer scenarios.

Understanding which dimensions each method normalizes — and why that matters — is the key to choosing the right normalization for your architecture.

Chapter Roadmap

Click any topic to jump in

Covariate Shift

Layer input distributions drift during training, causing slow convergence and init sensitivity.

solved by

BatchNorm

Normalizes per channel across batch and spatial dims — best for CNNs with large batches.

LayerNorm

Normalizes per sample across all features — batch-independent, standard in Transformers.

GroupNorm / InstanceNorm

Interpolates between LN and per-channel normalization — used in detection and style transfer.

batch-independent variant

Choosing Norm

Architecture and batch size determine the best choice; RMSNorm is the modern LLM default.

Normalization addresses a fundamental training challenge: the shifting distribution of layer inputs during training.

The Problem

During neural network training, a hidden problem emerges:

Each layer's input distribution changes as the preceding layers' parameters are updated
This is called internal covariate shift — later layers must constantly adapt to new input distributions
Forces the use of small learning rates (to avoid instability from distribution shifts)
Requires careful initialization (bad init → exploding/vanishing activations)
Training is slow because each layer is chasing a moving target

Without normalization, deeper networks become increasingly difficult to train because distribution shifts compound across layers.

The Solution

Normalization standardizes activations to have zero mean and unit variance:

$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$

Then applies learnable scale ( $\gamma$ ) and shift ( $\beta$ ):

$y = \gamma \hat{x} + \beta$

Why learnable parameters? If the optimal distribution isn't zero-mean unit-variance, the network can learn to undo the normalization via $\gamma$ and $\beta$ . The normalization provides a good starting point; the learned parameters give flexibility.

Benefits:

Stabilizes activation distributions across layers
Enables higher learning rates → faster training
Reduces sensitivity to weight initialization
Acts as a mild regularizer (due to batch statistics noise)

Key Points

Internal covariate shift: layer input distributions change during training

Normalization standardizes activations to zero mean, unit variance

Learnable $\gamma$ (scale) and $\beta$ (shift) preserve representational power

Enables higher learning rates and faster convergence

Mathematical Formulation

Normalization Formula

$y = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$

μ and σ² are computed over different dimensions depending on the normalization type

Mathematical Intuition

Normalization is a change-of-coordinates that makes the Hessian of the loss better conditioned. By standardizing activations to zero mean and unit variance, the loss landscape becomes more isotropic, so gradient descent sees smaller eigenvalue ratios — enabling larger stable learning rates.

Back to All Papers

Study Plans

ML Study Plan All Papers Practice Problems