Layer Normalization

easyFREETransformer ComponentsLLM 1: Foundations

Implement layer normalization for a single vector.

Layer normalization normalizes across the feature dimension: $\text{LayerNorm}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$

where μ and σ² are the mean and variance of x (variance over the n features, dividing by n), γ (gain) and β (bias) are learnable parameters.

For this problem, use γ = 1 and β = 0 (no affine transform), and ε = 1e-5.

Input: x, a 1-D NumPy array of floats (the vector x), and eps (default 1e-5) Output: The normalized vector as a NumPy array of the same length (the tests print each value rounded to 4 decimal places).

Example

Input

1.0 2.0 3.0

Output

-1.2247 0.0000 1.2247

Reasoning

Input: x is one feature vector, here $[1.0, 2.0, 3.0]$ (three features of a single token); eps is the small constant $\epsilon = 10^{-5}$ .
Mean: $\mu = \frac{1 + 2 + 3}{3} = 2$ .
Variance: average squared distance from the mean, dividing by the number of features $n = 3$ (not $n-1$ ): $\sigma^2 = \frac{(-1)^2 + 0^2 + 1^2}{3} = \frac{2}{3} \approx 0.6667$ .
Scale: $\sqrt{\sigma^2 + \epsilon} = \sqrt{0.66668} \approx 0.81650$ , so each centred value $x_i - \mu \in \{-1, 0, 1\}$ is divided by it: $\frac{\pm 1}{0.81650} \approx \pm 1.2247$ .
Output: a vector of the same length, -1.2247 0.0000 1.2247, with mean $0$ and variance just under $1$ ( $\epsilon$ shrinks it very slightly).
Notice the role of $\epsilon$ : for a constant vector like $[5, 5]$ the variance is $0$ , and $\epsilon$ keeps the denominator nonzero so the result is all zeros instead of undefined.

Constraints

1 <= dimension <= 100
Use ε = 1e-5 for numerical stability
γ = 1, β = 0
Round to 4 decimal places

Learn this in

Transformer ArchitectureLLM Study Plan Attention MechanismsNLP Study Plan

More from LLM 1: Foundations

Transformer Block Forward Passhard Feed-Forward Networkeasy KV-Cache Attentionhard Multi-Head Attentionmedium Causal Attention Maskmedium

Concept primer

Background Knowledge

Layer Normalization is a technique used in deep learning to normalize the input data for each layer. This is different from Batch Normalization, which normalizes the input data for each mini-batch. Layer normalization is particularly useful in Transformer architectures, where it helps to stabilize the training process and improve the model's performance. The key idea behind layer normalization is to normalize the input data across the feature dimension, which helps to reduce the effect of internal covariate shift.

The LayerNorm function takes a vector x as input and returns a normalized vector. The normalization process involves subtracting the mean μ of x and then dividing by the square root of the variance σ² of x plus a small constant ε. This process is similar to Standardization, which is a common technique used in data preprocessing. However, in layer normalization, the mean and variance are computed across the feature dimension, whereas in standardization, they are computed across the sample dimension.

The gain γ and bias β are learnable parameters that are used to scale and shift the normalized vector, respectively. In this problem, we are using γ = 1 and β = 0, which means that the normalized vector is not scaled or shifted. The small constant ε = 1e-5 is added to the variance to prevent division by zero.

Algorithm/Approach

The general approach to solving this problem is to follow the LayerNorm formula and implement it in code. The key steps involve computing the mean and variance of the input vector, normalizing the vector, and then scaling and shifting the normalized vector using the gain and bias parameters.

Step-by-Step Strategy

To implement the solution, follow these steps:

Compute the mean μ of the input vector x.
Compute the variance σ² of the input vector x.
Compute the normalized vector using the LayerNorm formula.
Round the normalized vector values to 4 decimal places.
Output the normalized vector.

Common Pitfalls

Some common pitfalls to watch out for when implementing the solution include:

Forgetting to add the small constant ε to the variance to prevent division by zero.
Using the wrong formula for computing the mean and variance.
Not rounding the normalized vector values to 4 decimal places.

Time & Space Complexity

The time complexity of the solution is O(n), where n is the length of the input vector, since we need to compute the mean and variance of the vector. The space complexity is also O(n), since we need to store the normalized vector. Note that the space complexity can be reduced to O(1) if we normalize the vector in-place, without storing the normalized vector separately.

solution.py

Python 3.13

Test results