Layer Normalization
Implement layer normalization for a single vector.
Layer normalization normalizes across the feature dimension: LayerNorm(x)=σ2+ϵ​x−μ​⋅γ+β
where μ and σ² are the mean and variance of x, γ (gain) and β (bias) are learnable parameters.
For this problem, use γ = 1 and β = 0 (no affine transform), and ε = 1e-5.
Input: Space-separated floats (the vector x) Output: Normalized vector, values rounded to 4 decimal places.
Example:
1.0 2.0 3.0
-1.2247 0.0000 1.2247
- The input vector x is [1.0,2.0,3.0].
- We calculate the mean μ and variance σ2 of x: μ=31.0+2.0+3.0​=2.0 and σ2=3(1.0−2.0)2+(2.0−2.0)2+(3.0−2.0)2​=32​.
- Then, we apply the layer normalization formula: LayerNorm(x)=σ2+ϵ​x−μ​⋅γ+β=32​+1e−5​x−2.0​⋅1+0.
- The final output is calculated by applying the formula to each element of x: [32​+1e−5​1.0−2.0​,32​+1e−5​2.0−2.0​,32​+1e−5​3.0−2.0​] = [−1.2247,0.0000,1.2247].
Constraints:
- 1 <= dimension <= 100
- Use ε = 1e-5 for numerical stability
- γ = 1, β = 0
- Round to 4 decimal places
Background Knowledge
Layer Normalization is a technique used in deep learning to normalize the input data for each layer. This is different from Batch Normalization, which normalizes the input data for each mini-batch. Layer normalization is particularly useful in Transformer architectures, where it helps to stabilize the training process and improve the model's performance. The key idea behind layer normalization is to normalize the input data across the feature dimension, which helps to reduce the effect of internal covariate shift.
The LayerNorm function takes a vector x as input and returns a normalized vector. The normalization process involves subtracting the mean μ of x and then dividing by the square root of the variance σ² of x plus a small constant ε. This process is similar to Standardization, which is a common technique used in data preprocessing. However, in layer normalization, the mean and variance are computed across the feature dimension, whereas in standardization, they are computed across the sample dimension.
The gain γ and bias β are learnable parameters that are used to scale and shift the normalized vector, respectively. In this problem, we are using γ = 1 and β = 0, which means that the normalized vector is not scaled or shifted. The small constant ε = 1e-5 is added to the variance to prevent division by zero.
Algorithm/Approach
The general approach to solving this problem is to follow the LayerNorm formula and implement it in code. The key steps involve computing the mean and variance of the input vector, normalizing the vector, and then scaling and shifting the normalized vector using the gain and bias parameters.
Step-by-Step Strategy
To implement the solution, follow these steps:
- Compute the mean μ of the input vector x.
- Compute the variance σ² of the input vector x.
- Compute the normalized vector using the LayerNorm formula.
- Round the normalized vector values to 4 decimal places.
- Output the normalized vector.
Common Pitfalls
Some common pitfalls to watch out for when implementing the solution include:
- Forgetting to add the small constant ε to the variance to prevent division by zero.
- Using the wrong formula for computing the mean and variance.
- Not rounding the normalized vector values to 4 decimal places.
Time & Space Complexity
The time complexity of the solution is O(n), where n is the length of the input vector, since we need to compute the mean and variance of the vector. The space complexity is also O(n), since we need to store the normalized vector. Note that the space complexity can be reduced to O(1) if we normalize the vector in-place, without storing the normalized vector separately.