Paper breakdownTrainingFREE2016

BatchNorm vs LayerNorm vs GroupNorm

Normalization techniques compared, plus InstanceNorm

Sergey Ioffe, Jimmy Lei Ba, Yuxin Wu, Dmitry Ulyanov

Concepts

Formulas

Demos

Paper Overview

Normalization layers are essential building blocks in modern deep learning. They stabilize training, enable higher learning rates, and reduce sensitivity to initialization — but different normalization types operate over different dimensions, making them suited for different architectures.

Batch Normalization (2015) normalizes across the batch dimension, making it ideal for CNNs with large batches. Layer Normalization (2016) normalizes across features per sample, becoming the standard for Transformers. Group Normalization (2018) and Instance Normalization (2016) offer alternatives for small-batch and style transfer scenarios.

Understanding which dimensions each method normalizes — and why that matters — is the key to choosing the right normalization for your architecture.

Chapter Roadmap

Click any topic to jump in

Interactive demo

Covariate Shift

Layer input distributions drift during training, causing slow convergence and init sensitivity.

solved by

BatchNorm

Normalizes per channel across batch and spatial dims — best for CNNs with large batches.

LayerNorm

Normalizes per sample across all features — batch-independent, standard in Transformers.

GroupNorm / InstanceNorm

Interpolates between LN and per-channel normalization — used in detection and style transfer.

batch-independent variant

Choosing Norm

Architecture and batch size determine the best choice; RMSNorm is the modern LLM default.

Concept by concept

5 sections, in reading order.

01 / 05Why Normalization? Internal Covariate Shift

Normalization addresses a fundamental training challenge: the shifting distribution of layer inputs during training.

The Problem

During neural network training, a hidden problem emerges:

Each layer's input distribution changes as the preceding layers' parameters are updated
This is called internal covariate shift — later layers must constantly adapt to new input distributions
Forces the use of small learning rates (to avoid instability from distribution shifts)
Requires careful initialization (bad init → exploding/vanishing activations)
Training is slow because each layer is chasing a moving target

Without normalization, deeper networks become increasingly difficult to train because distribution shifts compound across layers.

The Solution

Normalization standardizes activations to have zero mean and unit variance:

$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$

Then applies learnable scale ( $\gamma$ ) and shift ( $\beta$ ):

$y = \gamma \hat{x} + \beta$

Why learnable parameters? If the optimal distribution isn't zero-mean unit-variance, the network can learn to undo the normalization via $\gamma$ and $\beta$ . The normalization provides a good starting point; the learned parameters give flexibility.

Benefits:

Stabilizes activation distributions across layers
Enables higher learning rates → faster training
Reduces sensitivity to weight initialization
Acts as a mild regularizer (due to batch statistics noise)

Key Points

1
Internal covariate shift: layer input distributions change during training
2
Normalization standardizes activations to zero mean, unit variance
3
Learnable $\gamma$ (scale) and $\beta$ (shift) preserve representational power
4
Enables higher learning rates and faster convergence

Mathematical Formulation

Normalization Formula

$y = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$

μ and σ² are computed over different dimensions depending on the normalization type

Mathematical Intuition

Normalization is a change-of-coordinates that makes the Hessian of the loss better conditioned. By standardizing activations to zero mean and unit variance, the loss landscape becomes more isotropic, so gradient descent sees smaller eigenvalue ratios — enabling larger stable learning rates.

BatchNorm computes statistics across the batch dimension, making it the standard for CNNs with large batch sizes.

The Problem

The key question is: over which dimensions should we compute mean and variance?

For a 4D tensor with shape $(N, C, H, W)$ :

$N$ = batch size (number of samples)
$C$ = channels (feature maps)
$H, W$ = spatial dimensions (height, width)

We need statistics that capture the typical activation range for each feature, while being robust enough to generalize across samples.

The Solution

Batch Normalization computes statistics for each channel across the batch and spatial dimensions:

$\mu_c = \frac{1}{N \cdot H \cdot W} \sum_{n,h,w} x_{n,c,h,w}$ $\sigma_c^2 = \frac{1}{N \cdot H \cdot W} \sum_{n,h,w} (x_{n,c,h,w} - \mu_c)^2$

Each channel gets one mean and one variance computed from all samples in the batch and all spatial locations.

Training vs Inference:

Training: Uses mini-batch statistics (introduces noise → mild regularization)
Inference: Uses running mean/variance accumulated during training (no batch dependency)

This is why BatchNorm behaves differently at train vs eval time — and why you must call model.eval() before inference!

Key Points

1
Normalizes across $N, H, W$ for each channel $C$ independently
2
Each channel has its own $\mu_c, \sigma_c^2, \gamma_c, \beta_c$
3
Requires sufficiently large batch size (typically 16-32+)
4
Uses running statistics at inference (no batch dependency)
5
Most effective for CNNs with large, stable batch sizes

Mathematical Formulation

BatchNorm Statistics

$\mu_c = \frac{1}{NHW} \sum_{n=1}^{N} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{n,c,h,w}$

Mean computed across batch (N) and spatial (H,W) for each channel c

Mathematical Intuition

BatchNorm computes statistics $\mu_c, \sigma_c^2$ over $N \cdot H \cdot W$ samples per channel — a Monte Carlo estimate of the channel's population distribution. Variance of this estimate scales as $1/(NHW)$ , so small batches produce unreliable statistics and degrade performance.

Comparison

Before / Traditional

Training mode:

Statistics from current mini-batch
Adds stochastic noise (regularization effect)
Batch dependency between samples

After / This Paper

Inference mode (model.eval()):

Running mean/variance from training
Deterministic output
No batch dependency — can process single samples

LayerNorm normalizes across all features for each sample independently, making it batch-size independent and ideal for Transformers.

The Problem

BatchNorm has critical limitations:

Batch size dependency: Small batches (1-4) give noisy statistics → poor performance
Batch dependency: Each sample's normalization depends on other samples in the batch
RNNs/Transformers: Variable sequence lengths make batch statistics unreliable
Distributed training: Requires synchronized statistics across GPUs
Inference: Must maintain running statistics; single-sample inference uses different code path

These issues make BatchNorm unsuitable for Transformers, RNNs, and any architecture that processes variable-length sequences or uses small batches.

The Solution

Layer Normalization normalizes across all features for each sample independently:

$\mu_n = \frac{1}{C \cdot H \cdot W} \sum_{c,h,w} x_{n,c,h,w}$

Each sample gets its own mean and variance, computed from all its features. No dependency on other samples in the batch!

In Transformers (most common use), for each token with embedding dimension $d$ : $\mu = \frac{1}{d} \sum_{i=1}^{d} x_i, \quad \sigma^2 = \frac{1}{d} \sum_{i=1}^{d} (x_i - \mu)^2$

Advantages:

Works with any batch size, including batch=1
No running statistics needed — same behavior at train and inference
Each sample normalized independently — no cross-sample dependency
Perfect for variable-length sequences in Transformers

Key Points

1
Normalizes across $C, H, W$ for each sample $N$ independently
2
Batch-size independent — works with batch=1
3
Same behavior at training and inference (no running statistics)
4
Default normalization for Transformers (BERT, GPT, ViT)
5
In Transformers: normalizes across embedding dimension per token

Mathematical Formulation

LayerNorm Statistics

$\mu_n = \frac{1}{CHW} \sum_{c=1}^{C} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{n,c,h,w}$

Mean computed across all features (C,H,W) for each sample n independently

Mathematical Intuition

LayerNorm's statistics live entirely within a single sample: $\mu, \sigma^2$ over features. This removes the cross-sample dependency, making the computation equivariant to batch size and identical at train and inference. Perfect for sequences where batch composition is arbitrary.

GroupNorm and InstanceNorm provide alternatives that interpolate between BatchNorm and LayerNorm, suited for specific architectures.

The Problem

Neither BatchNorm nor LayerNorm is optimal for all scenarios:

Object detection/segmentation: Uses small batches (1-2 images per GPU due to high-resolution inputs) → BatchNorm fails
Style transfer: Per-channel statistics encode style information → need per-instance, per-channel normalization
LayerNorm for CNNs: Normalizing ALL channels together ignores that different channels represent different features (edges, textures, objects)

Is there a middle ground between normalizing all channels together (LayerNorm) and each channel separately?

The Solution

Group Normalization divides channels into $G$ groups and normalizes within each group:

$\mu_{n,g} = \frac{1}{(C/G) \cdot H \cdot W} \sum_{c \in \text{group}_g} \sum_{h,w} x_{n,c,h,w}$

$G=1$ : equivalent to LayerNorm (all channels in one group)
$G=C$ : equivalent to InstanceNorm (each channel is its own group)
Typical $G=32$ for detection/segmentation

Instance Normalization ( $G = C$ ) normalizes each channel independently per sample:

$\mu_{n,c} = \frac{1}{H \cdot W} \sum_{h,w} x_{n,c,h,w}$

Used in style transfer because per-channel statistics encode style information (e.g., color palette, texture patterns). Removing these statistics via normalization enables style manipulation.

Adaptive Instance Normalization (AdaIN) goes further: it replaces the learned $\gamma, \beta$ with statistics from a style image, enabling arbitrary style transfer.

Key Points

1
GroupNorm: channels split into $G$ groups, normalized per group per sample
2
InstanceNorm: special case with $G=C$ (each channel independent)
3
GroupNorm works with any batch size — used in detection frameworks (Detectron2)
4
InstanceNorm used in style transfer (per-channel stats encode style)
5
Typical GroupNorm: $G=32$ groups

Mathematical Formulation

Group Normalization

$\mu_{n,g} = \frac{1}{|S_g|} \sum_{k \in S_g} x_{n,k}, \quad S_g = \{c : \lfloor c \cdot G/C \rfloor = g\}$

S_g is the set of channel indices in group g; statistics computed per group per sample

Instance Normalization

$\mu_{n,c} = \frac{1}{HW} \sum_{h=1}^{H} \sum_{w=1}^{W} x_{n,c,h,w}$

Each channel normalized independently for each sample (GroupNorm with G=C)

Mathematical Intuition

GroupNorm is a smooth interpolation between LayerNorm ( $G = 1$ ) and InstanceNorm ( $G = C$ ). Choosing $G$ trades off feature correlation (fewer groups = more features normalized together) against flexibility (more groups = finer-grained statistics).

Comparison

Before / Traditional

BatchNorm at small batch sizes:

Batch=2: noisy statistics, ~3% accuracy drop
Batch=1: completely fails (zero variance)
Degrades detection/segmentation performance

After / This Paper

GroupNorm at small batch sizes:

Consistent performance across all batch sizes
Only ~0.5% below BatchNorm at batch=32
Default in Detectron2 for object detection

Each normalization type excels in specific scenarios. Here's a practical guide for choosing the right one.

The Problem

With four normalization types available, the choice can be overwhelming:

Which normalization for Transformers?
What about CNNs with small batches?
Does it matter for inference speed?
What about newer alternatives like RMSNorm?

The Solution

Decision guide by architecture:

Architecture	Recommended	Why
CNNs (large batch)	BatchNorm	Best accuracy, mature implementation
Transformers	LayerNorm	Batch-independent, handles variable sequences
Detection/Segmentation	GroupNorm	Small batches (1-2), stable performance
Style Transfer	InstanceNorm	Per-channel stats encode style
LLMs (LLaMA, Gemma)	RMSNorm	Faster than LayerNorm, similar accuracy

Modern trends:

Pre-Norm vs Post-Norm: Placing LayerNorm before attention (Pre-Norm) is more stable for training very deep Transformers
RMSNorm: Simplified LayerNorm that only divides by RMS (no mean subtraction): $\hat{x} = x / \text{RMS}(x)$ . Used in LLaMA, Gemma. ~5% faster than LayerNorm
No normalization: Some architectures (e.g., NFNets) replace normalization with careful initialization and scaling

Key Points

1
BatchNorm: CNNs with batch ≥ 16, uses running stats at inference
2
LayerNorm: Transformers, RNNs, batch-independent
3
GroupNorm: Detection/segmentation, small batches, stable across sizes
4
InstanceNorm: Style transfer, per-channel normalization
5
RMSNorm: Faster LayerNorm variant used in modern LLMs

Mathematical Formulation

RMSNorm (Modern Alternative)

$\hat{x}_i = \frac{x_i}{\text{RMS}(x)} \cdot \gamma_i, \quad \text{RMS}(x) = \sqrt{\frac{1}{d} \sum_{i=1}^{d} x_i^2}$

Simplified LayerNorm: no mean subtraction, only RMS scaling. Used in LLaMA, Gemma

Mathematical Intuition

The dimensions of normalization encode assumptions about data structure: BN assumes batch samples are i.i.d. from the same distribution; LN assumes features within a sample are comparable; GN assumes channel groups share statistics. RMSNorm drops the mean subtraction entirely — for post-softmax distributions, centering is redundant.

Go deeper

Study Plans

ML Study Plan All Papers Practice Problems

Paper Overview

Chapter Roadmap

Covariate Shift

BatchNorm

LayerNorm

GroupNorm / InstanceNorm

Choosing Norm

Concept by concept

Why Normalization? Internal Covariate Shift

The Problem

The Solution

Key Points

Mathematical Formulation

Normalization Formula

Mathematical Intuition

Batch Normalization

The Problem

The Solution

Key Points

Mathematical Formulation

BatchNorm Statistics

Mathematical Intuition

Comparison

Before / Traditional

After / This Paper

Layer Normalization

The Problem

The Solution

Key Points

Mathematical Formulation

LayerNorm Statistics

Mathematical Intuition

Group Normalization and Instance Normalization

The Problem

The Solution

Key Points

Mathematical Formulation

Group Normalization

Instance Normalization

Mathematical Intuition

Comparison

Before / Traditional

After / This Paper

Choosing the Right Normalization

The Problem

The Solution

Key Points

Mathematical Formulation

RMSNorm (Modern Alternative)

Mathematical Intuition

Further Reading

Related Papers

Study Plans