MLP Projector Parameter Count

easyPROBridge Modules: Projectors, Q-Former & Cross-AttentionVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

LLaVA-1.5 replaced its single linear projector with a two-layer MLP that maps vision features into the language model's embedding space. Count the trainable parameters of that projector.

Background

A two-layer MLP projector is Linear(d_in, d_hidden) -> GELU -> Linear(d_hidden, d_out). A Linear(a, b) layer with bias has a*b + b parameters (weights plus one bias per output). GELU has none. So the total is

$(d_{in}\, d_{hidden} + d_{hidden}) + (d_{hidden}\, d_{out} + d_{out})$

Your Task

Implement:

def projector_params(d_in, d_hidden, d_out, bias=True):

Return the total parameter count as an int. When bias is false, drop the bias terms.

Input Format

d_in, d_hidden, d_out (int): layer widths.
bias (bool): whether the Linear layers have biases.

Output Format

A single int.

Sample

print(projector_params(1024, 4096, 4096))

Output:

20979712

Example

Input

print(projector_params(1024, 4096, 4096))

Output

20979712

Reasoning

Identify the layer dimensions and bias setting: $d_{in} = 1024$ , $d_{hidden} = 4096$ , $d_{out} = 4096$ , and bias is True (default), so bias terms are included.
Calculate the parameter count for the first linear layer ( $d_{in} \to d_{hidden}$ ): weights are $1024 \times 4096 = 4{,}194{,}304$ and bias is $4096$ , totaling $4{,}198{,}400$ .
Calculate the parameter count for the second linear layer ( $d_{hidden} \to d_{out}$ ): weights are $4096 \times 4096 = 16{,}777{,}216$ and bias is $4096$ , totaling $16{,}781{,}312$ .
Sum the parameters from both layers to get the total projector size: $4{,}198{,}400 + 16{,}781{,}312 = 20{,}979{,}712$ .
The final output is 20979712

Constraints

1 <= d_in, d_hidden, d_out <= 100000.
Each Linear contributes in*out weights plus out biases when bias.
Return an int.

Learn this in

Multimodal LLMsLLM Study Plan Introduction to Vision-Language ModelsVLM Study Plan Vision-Language AlignmentVLM Study Plan Instruction Tuning & the LLaVA ParadigmVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

In Vision-Language Models (VLMs) like LLaVA, a projector bridges the vision encoder and the language model. The vision encoder produces feature vectors of dimension $d_{in}$ , while the language model expects token embeddings of dimension $d_{out}$ . The projector maps between these spaces. LLaVA-1.5 uses a two-layer MLP (Multi-Layer Perceptron) with a GELU activation in between, rather than a single linear layer, to allow non-linear transformation of visual features.

A Linear layer (fully connected layer) maps input of size $a$ to output of size $b$ . It contains a weight matrix of shape $(b, a)$ and, optionally, a bias vector of shape $(b,)$ . The total parameter count for one Linear layer is therefore $a \times b$ (weights) plus $b$ (biases) if bias is enabled. The GELU activation function is element-wise and introduces no additional trainable parameters.

This problem is a straightforward parameter counting exercise. You are not implementing the forward pass or training; you are simply summing the number of trainable parameters across the two Linear layers in the MLP projector.

2. Algorithm Approach

The approach is direct arithmetic computation. There is no iteration, search, or optimization involved. You:

Compute the parameter count for the first Linear layer: $\text{Linear}(d_{in}, d_{hidden})$ .
Compute the parameter count for the second Linear layer: $\text{Linear}(d_{hidden}, d_{out})$ .
Sum the two counts.
Conditionally include or exclude bias terms based on the bias flag.

The formula for a single Linear layer with dimensions $(a, b)$ is:

With bias: $a \cdot b + b$
Without bias: $a \cdot b$

Since GELU contributes zero parameters, the total is simply the sum of the two Linear layers' counts.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.