📘

MLP Projector Parameter Count

EasyVision-Language Models 📚 VLM 1: Vision-Language Models

Problem Statement

LLaVA-1.5 replaced its single linear projector with a two-layer MLP that maps vision features into the language model's embedding space. Count the trainable parameters of that projector.

Background

A two-layer MLP projector is Linear(d_in, d_hidden) -> GELU -> Linear(d_hidden, d_out). A Linear(a, b) layer with bias has a*b + b parameters (weights plus one bias per output). GELU has none. So the total is

$(d_{in}\, d_{hidden} + d_{hidden}) + (d_{hidden}\, d_{out} + d_{out})$

Your Task

Implement:

def projector_params(d_in, d_hidden, d_out, bias=True):

Return the total parameter count as an int. When bias is false, drop the bias terms.

Input Format

d_in, d_hidden, d_out (int): layer widths.
bias (bool): whether the Linear layers have biases.

Output Format

A single int.

Sample

print(projector_params(1024, 4096, 4096))

Output:

20979712

Example:

Input:

print(projector_params(1024, 4096, 4096))

Output:

20979712

Reasoning:

Identify the layer dimensions and bias setting: $d_{in} = 1024$ , $d_{hidden} = 4096$ , $d_{out} = 4096$ , and bias is True (default), so bias terms are included.
Calculate the parameter count for the first linear layer ( $d_{in} \to d_{hidden}$ ): weights are $1024 \times 4096 = 4{,}194{,}304$ and bias is $4096$ , totaling $4{,}198{,}400$ .
Calculate the parameter count for the second linear layer ( $d_{hidden} \to d_{out}$ ): weights are $4096 \times 4096 = 16{,}777{,}216$ and bias is $4096$ , totaling $16{,}781{,}312$ .
Sum the parameters from both layers to get the total projector size: $4{,}198{,}400 + 16{,}781{,}312 = 20{,}979{,}712$ .
The final output is 20979712

Constraints:

1 <= d_in, d_hidden, d_out <= 100000.
Each Linear contributes in*out weights plus out biases when bias.
Return an int.

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention MassHard CHAIR-i and CHAIR-s Hallucination RatesMedium Merge a LoRA Update into the Base WeightMedium QLoRA Memory Footprint EstimateMedium POPE Precision, Recall, and F1Medium

1. Background Knowledge

In Vision-Language Models (VLMs) like LLaVA, a projector bridges the vision encoder and the language model. The vision encoder produces feature vectors of dimension $d_{in}$ , while the language model expects token embeddings of dimension $d_{out}$ . The projector maps between these spaces. LLaVA-1.5 uses a two-layer MLP (Multi-Layer Perceptron) with a GELU activation in between, rather than a single linear layer, to allow non-linear transformation of visual features.

A Linear layer (fully connected layer) maps input of size $a$ to output of size $b$ . It contains a weight matrix of shape $(b, a)$ and, optionally, a bias vector of shape $(b,)$ . The total parameter count for one Linear layer is therefore $a \times b$ (weights) plus $b$ (biases) if bias is enabled. The GELU activation function is element-wise and introduces no additional trainable parameters.

This problem is a straightforward parameter counting exercise. You are not implementing the forward pass or training; you are simply summing the number of trainable parameters across the two Linear layers in the MLP projector.

2. Algorithm Approach

The approach is direct arithmetic computation. There is no iteration, search, or optimization involved. You:

Compute the parameter count for the first Linear layer: $\text{Linear}(d_{in}, d_{hidden})$ .
Compute the parameter count for the second Linear layer: $\text{Linear}(d_{hidden}, d_{out})$ .
Sum the two counts.
Conditionally include or exclude bias terms based on the bias flag.

The formula for a single Linear layer with dimensions $(a, b)$ is:

With bias: $a \cdot b + b$
Without bias: $a \cdot b$

Since GELU contributes zero parameters, the total is simply the sum of the two Linear layers' counts.

3. Step-by-Step Strategy

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.