Q-Former Output Shape and Parameter Count

easyFREEBridge Modules: Projectors, Q-Former & Cross-AttentionVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Size a BLIP-2 style Q-Former: given the number of learned query tokens, the widths involved and the number of cross-attention blocks, report the output shape, the parameter count, and the compression ratio it achieves over the raw visual tokens.

Background

A linear projector maps every one of the vision encoder's tokens into the LLM, so a 257-token ViT costs 257 LLM positions. A Q-Former breaks that coupling: it owns a fixed set of num_queries learned query embeddings (32 in BLIP-2) that cross-attend to the image features and emit exactly num_queries output tokens - no matter how many image tokens came in. That is the compression ratio:

compression = num_image_tokens / num_queries

Count the parameters with this simplified model:

Learned queries: num_queries * d_query
Each of num_layers cross-attention blocks has four projections, all with a bias:
- W_Q: d_query x d_query + d_query
- W_K: d_image x d_query + d_query
- W_V: d_image x d_query + d_query
- W_O: d_query x d_query + d_query
Final projector into the LLM: d_query x d_llm + d_llm

W_K and W_V read the frozen vision encoder's width d_image (1408 for EVA-ViT-g) and write the Q-Former's own width d_query (768), which is why the two widths appear asymmetrically.

Notice what is missing from the list: num_image_tokens. Sequence length never appears in a parameter count - only in the activation/FLOP cost. The Q-Former's size is identical whether it consumes 257 or 4097 image tokens.

Your Task

Implement:

def qformer_stats(num_queries, d_query, d_image, num_image_tokens, num_layers, d_llm):

Return [num_image_tokens, num_queries, d_llm, total_params, compression] where the first four are ints and compression is num_image_tokens / num_queries rounded to 4 decimals.

Input Format

Six positive integers, as named above.

Output Format

[int, int, int, int, float]

Sample

print(qformer_stats(32, 768, 1408, 256, 12, 4096))

Output:

[256, 32, 4096, 43319296, 8.0]

Example

Input

print(qformer_stats(32, 768, 1408, 256, 12, 4096))

Output

[256, 32, 4096, 43319296, 8.0]

Reasoning

Inputs: 32 learned queries of width $d_{query}=768$ , a vision encoder of width $d_{image}=1408$ emitting 256 tokens, 12 cross-attention blocks, and an LLM of width $d_{llm}=4096$ .
One block: the two $768 \times 768$ projections cost $768^2 + 768 = 590{,}592$ each, the two $1408 \times 768$ ones cost $1{,}081{,}344 + 768 = 1{,}082{,}112$ each, so a block holds $2(590{,}592) + 2(1{,}082{,}112) = 3{,}345{,}408$ parameters. Twelve blocks give $40{,}144{,}896$ .
The rest: the queries add $32 \times 768 = 24{,}576$ and the projector adds $768 \times 4096 + 4096 = 3{,}149{,}824$ , for a total of $43{,}319{,}296$ .
Compression: $256 / 32 = 8$ , so the output is [256, 32, 4096, 43319296, 8.0]: 256 visual tokens in, 32 tokens of width 4096 out.
Notice the token count only enters the ratio. With 4096 image tokens the total stays $43{,}319{,}296$ while the ratio jumps to $128$ .

Constraints

All inputs are positive integers
Every projection carries a bias vector of length equal to its output width
W_K and W_V map d_image -> d_query; W_Q and W_O map d_query -> d_query
The parameter count must NOT depend on num_image_tokens
Round compression to 4 decimals; return the counts as plain Python ints

Learn this in

Vision-Language AlignmentVLM Study Plan Instruction Tuning & the LLaVA ParadigmVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primer

1. Background Knowledge

In Vision-Language Models (VLMs) like BLIP-2, bridging the gap between a Vision Encoder (e.g., ViT) and a Large Language Model (LLM) is critical. A naive approach uses a linear projector that maps every visual token directly to the LLM embedding space. This is inefficient because vision encoders often produce hundreds of tokens (e.g., 257 for a 224x224 image), which consumes significant context window space in the LLM.

The Q-Former (Query Transformer) solves this by introducing a fixed set of learned query tokens (num_queries). Instead of processing all image tokens, the Q-Former uses these queries to cross-attend to the image features. This mechanism allows the model to distill the most relevant visual information into a compact, fixed-size representation, regardless of the input image resolution or the number of visual tokens generated by the encoder.

The parameter count of a Q-Former is determined by its internal architecture, not the input sequence length. It consists of:

Learned Queries: Embeddings for the query tokens.
Cross-Attention Blocks: Each block contains projections for Query ( $W_Q$ ), Key ( $W_K$ ), Value ( $W_V$ ), and Output ( $W_O$ ). Note that $W_K$ and $W_V$ project from the image dimension ( $d_{image}$ ) to the query dimension ( $d_{query}$ ), while $W_Q$ and $W_O$ operate within the query dimension.
Final Projector: A linear layer mapping the final query outputs to the LLM's embedding dimension ( $d_{llm}$ ).

2. Algorithm Approach

The problem requires calculating the output shape, total parameter count, and compression ratio of a Q-Former module. The approach is purely arithmetic based on the provided architectural specifications.

Output Shape: The Q-Former always outputs num_queries tokens, each with dimension d_llm after the final projection. The first dimension of the output tensor is typically the batch size, but here we focus on the sequence and feature dimensions. The problem asks for [num_image_tokens, num_queries, d_llm,...], implying we report the input sequence length, the fixed output sequence length, and the output feature dimension.
Parameter Count: Sum the parameters from all components:

Queries: num_queries * d_query
Per Layer: Calculate parameters for $W_Q, W_K, W_V, W_O$ including their biases. Multiply by num_layers.
Final Projector: d_query * d_llm + bias.

Compression Ratio: Calculate num_image_tokens / num_queries.

3. Step-by-Step Strategy

Define Variables: Extract num_queries, d_query, d_image, num_image_tokens, num_layers, and d_llm from the input.
Calculate Query Parameters:

param_queries = num_queries * d_query

Calculate Per-Layer Parameters:

$W_Q$ : d_query * d_query (weights) + d_query (bias)
$W_K$ : d_image * d_query (weights) + d_query (bias)
$W_V$ : d_image * d_query (weights) + d_query (bias)
$W_O$ : d_query * d_query (weights) + d_query (bias)
Sum these four values to get params_per_layer.
Multiply by num_layers to get total_layer_params.

Calculate Final Projector Parameters:

param_projector = d_query * d_llm + d_llm

Sum Total Parameters:

total_params = param_queries + total_layer_params + param_projector

Calculate Compression Ratio:

compression = num_image_tokens / num_queries
Round to 4 decimal places.

Return Result:

Return the list [num_image_tokens, num_queries, d_llm, total_params, compression].

4. Common Pitfalls

Bias Terms: The problem explicitly states that all projections have a bias. Forgetting to add the bias terms (+ d_query or + d_llm) will result in an incorrect parameter count.
Asymmetric Dimensions: Ensure $W_K$ and $W_V$ use d_image as the input dimension and d_query as the output dimension. $W_Q$ and $W_O$ use d_query for both input and output. Confusing these will lead to significant errors.
Sequence Length Independence: Remember that num_image_tokens does not affect the parameter count. It only affects the compression ratio and the output shape's first element (as reported in the list).
Rounding: The compression ratio must be rounded to 4 decimal places. Use Python's round() function correctly.
Integer vs. Float: The first four elements of the output list must be integers. The last element (compression) must be a float.

5. Time & Space Complexity

Time Complexity: $O(1)$ . The calculation involves a fixed number of arithmetic operations regardless of the input values. The complexity does not depend on the size of the tensors or the number of tokens, only on the scalar inputs.
Space Complexity: $O(1)$ . We only store a few integer and float variables to compute the result. No additional data structures are needed that scale with input size.

solution.py

Python 3.13

Test results