Loading...
Implement the forward pass of batch normalization using PyTorch tensors.
Batch normalization normalizes activations across the batch dimension:
x^i=σB2+ϵxi−μB yi=γx^i+β
Where:
Also track running mean/variance for inference mode.
x = [[1, 2], [3, 4], [5, 6]] # batch=3, features=2 gamma = [1, 1] beta = [0, 0] training = True
[[-1.22, -1.22], [0, 0], [1.22, 1.22]]
Feature 1: values [1,3,5], mean=3, std=1.63 Normalized: [-1.22, 0, 1.22]
Feature 2: values [2,4,6], mean=4, std=1.63 Normalized: [-1.22, 0, 1.22]
With gamma=1, beta=0: output equals normalized values.