Loading...
BatchNorm, LayerNorm, GroupNorm & InstanceNorm
Sergey Ioffe, Jimmy Lei Ba, Yuxin Wu, Dmitry Ulyanov
Read the Paper on arXivNormalization layers are essential building blocks in modern deep learning. They stabilize training, enable higher learning rates, and reduce sensitivity to initialization — but different normalization types operate over different dimensions, making them suited for different architectures.
Batch Normalization (2015) normalizes across the batch dimension, making it ideal for CNNs with large batches. Layer Normalization (2016) normalizes across features per sample, becoming the standard for Transformers. Group Normalization (2018) and Instance Normalization (2016) offer alternatives for small-batch and style transfer scenarios.
Understanding which dimensions each method normalizes — and why that matters — is the key to choosing the right normalization for your architecture.
Click any topic to jump in
Layer input distributions drift during training, causing slow convergence and init sensitivity.
Normalizes per channel across batch and spatial dims — best for CNNs with large batches.
Normalizes per sample across all features — batch-independent, standard in Transformers.
Interpolates between LN and per-channel normalization — used in detection and style transfer.
Architecture and batch size determine the best choice; RMSNorm is the modern LLM default.
Normalization addresses a fundamental training challenge: the shifting distribution of layer inputs during training.
During neural network training, a hidden problem emerges:
Without normalization, deeper networks become increasingly difficult to train because distribution shifts compound across layers.
Normalization standardizes activations to have zero mean and unit variance:
Then applies learnable scale () and shift ():
Why learnable parameters? If the optimal distribution isn't zero-mean unit-variance, the network can learn to undo the normalization via and . The normalization provides a good starting point; the learned parameters give flexibility.
Benefits:
Internal covariate shift: layer input distributions change during training
Normalization standardizes activations to zero mean, unit variance
Learnable (scale) and (shift) preserve representational power
Enables higher learning rates and faster convergence
μ and σ² are computed over different dimensions depending on the normalization type
Normalization is a change-of-coordinates that makes the Hessian of the loss better conditioned. By standardizing activations to zero mean and unit variance, the loss landscape becomes more isotropic, so gradient descent sees smaller eigenvalue ratios — enabling larger stable learning rates.