📘

RMSNorm Before the Projector

MediumVision-Language Models 📚 VLM 1: Vision-Language Models

Problem Statement

Modern VLMs RMS-normalize vision features before projecting them into the LM. Implement RMSNorm with a learnable per-feature scale.

Background

RMSNorm (Zhang & Sennrich, 2019) rescales a vector by its root-mean-square, with no mean subtraction:

$\text{RMSNorm}(x)_i = \frac{x_i}{\sqrt{\tfrac{1}{D}\sum_j x_j^2 + \epsilon}}\; g_i$

where g is a learned gain vector (length D) and \epsilon guards against division by zero. Unlike LayerNorm it keeps the direction's sign and is cheaper — which is why LLaMA-family LMs and their VLM wrappers use it.

Your Task

Implement:

def rmsnorm(x, g, eps=1e-6):

Return the normalized vector as a list rounded to 4 decimals.

Input Format

x: list of D floats.
g: list of D gains.
eps (float).

Output Format

A list of D floats rounded to 4 decimals.

Sample

print(rmsnorm([1.0, 2.0, 2.0, 4.0], [1.0, 1.0, 1.0, 1.0]))

Output:

[0.4, 0.8, 0.8, 1.6]

Example:

Input:

print(rmsnorm([1.0, 2.0, 2.0, 4.0], [1.0, 1.0, 1.0, 1.0]))

Output:

[0.4, 0.8, 0.8, 1.6]

Reasoning:

Compute the mean square of the input vector $x = [1.0, 2.0, 2.0, 4.0]$ to determine the scaling factor: $\text{ms} = \frac{1^2 + 2^2 + 2^2 + 4^2}{4} = \frac{1 + 4 + 4 + 16}{4} = \frac{25}{4} = 6.25$ .
Calculate the normalization denominator by adding the epsilon $\epsilon = 10^{-6}$ to the mean square and taking the square root: $\sqrt{6.25 + 0.000001} \approx \sqrt{6.250001} \approx 2.5000002$ .
Divide each element of $x$ by this denominator to normalize the vector, preserving the original direction: $[1.0/2.5, 2.0/2.5, 2.0/2.5, 4.0/2.5] \approx [0.4, 0.8, 0.8, 1.6]$ .
Multiply the normalized values by the gain vector $g = [1.0, 1.0, 1.0, 1.0]$ ; since all gains are 1, the values remain unchanged: $[0.4 \cdot 1, 0.8 \cdot 1, 0.8 \cdot 1, 1.6 \cdot 1] = [0.4, 0.8, 0.8, 1.6]$ .
The final output is [0.4, 0.8, 0.8, 1.6]

Constraints:

len(x) == len(g), 1 <= D <= 8192.
Divide by sqrt(mean(x^2) + eps); no mean subtraction.
Multiply by the gain elementwise; round to 4 decimals; avoid -0.0.

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention MassHard CHAIR-i and CHAIR-s Hallucination RatesMedium Merge a LoRA Update into the Base WeightMedium QLoRA Memory Footprint EstimateMedium POPE Precision, Recall, and F1Medium

1. Background Knowledge

RMSNorm is a normalization technique introduced by Zhang & Sennrich (2019) that stabilizes training in deep transformers by rescaling each feature vector to have a unit root-mean-square magnitude. Unlike LayerNorm, which subtracts the mean before dividing by the standard deviation, RMSNorm skips the centering step entirely. This preserves the original sign and direction of the vector while still controlling its scale, making it computationally cheaper (one fewer reduction pass) and empirically effective in LLaMA-family models and their vision-language wrappers.

In a VLM pipeline, vision encoders (e.g., SigLIP, CLIP) produce a set of patch-level feature vectors. Before these features are projected into the language model's embedding space via a projector (a linear layer or Q-Former), they are typically RMS-normalized. This ensures that the magnitude of visual tokens is consistent regardless of the input image's brightness or contrast, allowing the projector to focus on learning meaningful cross-modal mappings rather than compensating for scale differences.

The formula is:

\text{RMSNorm}(x)_i = \frac{x_i}{\sqrt{\frac{1}{D}\sum_{j=1}^{D} x_j^2 + \epsilon}} \cdot g_i

where $D$ is the feature dimension, $\epsilon$ is a small constant (e.g., $10^{-6}$ ) to prevent division by zero, and $g$ is a learnable gain vector of length $D$ initialized to ones. The gain allows the model to re-scale individual features after normalization, recovering expressiveness lost by the fixed RMS constraint.

2. Algorithm Approach

The implementation follows a straightforward element-wise transformation pattern with a single reduction step:

Compute the root-mean-square (RMS) of the input vector $x$ .
Divide each element of $x$ by the RMS value (plus $\epsilon$ under the square root).
Multiply each normalized element by the corresponding gain $g_i$ .
Round the result to the required precision.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.