Q-Former Output Shape and Parameter Count

1. Background Knowledge

In Vision-Language Models (VLMs) like BLIP-2, bridging the gap between a Vision Encoder (e.g., ViT) and a Large Language Model (LLM) is critical. A naive approach uses a linear projector that maps every visual token directly to the LLM embedding space. This is inefficient because vision encoders often produce hundreds of tokens (e.g., 257 for a 224x224 image), which consumes significant context window space in the LLM.

The Q-Former (Query Transformer) solves this by introducing a fixed set of learned query tokens (num_queries). Instead of processing all image tokens, the Q-Former uses these queries to cross-attend to the image features. This mechanism allows the model to distill the most relevant visual information into a compact, fixed-size representation, regardless of the input image resolution or the number of visual tokens generated by the encoder.

The parameter count of a Q-Former is determined by its internal architecture, not the input sequence length. It consists of:

1. Learned Queries: Embeddings for the query tokens.
2. Cross-Attention Blocks: Each block contains projections for Query ($W_Q$), Key ($W_K$), Value ($W_V$), and Output ($W_O$). Note that $W_K$ and $W_V$ project from the image dimension ($d_{image}$) to the query dimension ($d_{query}$), while $W_Q$ and $W_O$ operate within the query dimension.
3. Final Projector: A linear layer mapping the final query outputs to the LLM's embedding dimension ($d_{llm}$).

2. Algorithm Approach

The problem requires calculating the output shape, total parameter count, and compression ratio of a Q-Former module. The approach is purely arithmetic based on the provided architectural specifications.

1. Output Shape: The Q-Former always outputs num_queries tokens, each with dimension d_llm after the final projection. The first dimension of the output tensor is typically the batch size, but here we focus on the sequence and feature dimensions. The problem asks for [num_image_tokens, num_queries, d_llm,...], implying we report the input sequence length, the fixed output sequence length, and the output feature dimension.
2. Parameter Count: Sum the parameters from all components:

• Queries: num_queries * d_query
• Per Layer: Calculate parameters for $W_Q, W_K, W_V, W_O$ including their biases. Multiply by num_layers.
• Final Projector: d_query * d_llm + bias.

3. Compression Ratio: Calculate num_image_tokens / num_queries.

3. Step-by-Step Strategy

1. Define Variables: Extract num_queries, d_query, d_image, num_image_tokens, num_layers, and d_llm from the input.
2. Calculate Query Parameters:

• param_queries = num_queries * d_query

3. Calculate Per-Layer Parameters:

• $W_Q$: d_query * d_query (weights) + d_query (bias)
• $W_K$: d_image * d_query (weights) + d_query (bias)
• $W_V$: d_image * d_query (weights) + d_query (bias)
• $W_O$: d_query * d_query (weights) + d_query (bias)
• Sum these four values to get params_per_layer.
• Multiply by num_layers to get total_layer_params.

4. Calculate Final Projector Parameters:

• param_projector = d_query * d_llm + d_llm

5. Sum Total Parameters:

• total_params = param_queries + total_layer_params + param_projector

6. Calculate Compression Ratio:

• compression = num_image_tokens / num_queries
• Round to 4 decimal places.

7. Return Result:

• Return the list [num_image_tokens, num_queries, d_llm, total_params, compression].

4. Common Pitfalls

• Bias Terms: The problem explicitly states that all projections have a bias. Forgetting to add the bias terms (+ d_query or + d_llm) will result in an incorrect parameter count.
• Asymmetric Dimensions: Ensure $W_K$ and $W_V$ use d_image as the input dimension and d_query as the output dimension. $W_Q$ and $W_O$ use d_query for both input and output. Confusing these will lead to significant errors.
• Sequence Length Independence: Remember that num_image_tokens does not affect the parameter count. It only affects the compression ratio and the output shape's first element (as reported in the list).
• Rounding: The compression ratio must be rounded to 4 decimal places. Use Python's round() function correctly.
• Integer vs. Float: The first four elements of the output list must be integers. The last element (compression) must be a float.

5. Time & Space Complexity

• Time Complexity: $O(1)$. The calculation involves a fixed number of arithmetic operations regardless of the input values. The complexity does not depend on the size of the tensors or the number of tokens, only on the scalar inputs.
• Space Complexity: $O(1)$. We only store a few integer and float variables to compute the result. No additional data structures are needed that scale with input size.