Q-Former Output Shape and Parameter Count

Problem Statement

Size a BLIP-2 style Q-Former: given the number of learned query tokens, the widths involved and the number of cross-attention blocks, report the output shape, the parameter count, and the compression ratio it achieves over the raw visual tokens.

Background

A linear projector maps every one of the vision encoder's tokens into the LLM, so a 257-token ViT costs 257 LLM positions. A Q-Former breaks that coupling: it owns a fixed set of num_queries learned query embeddings (32 in BLIP-2) that cross-attend to the image features and emit exactly num_queries output tokens - no matter how many image tokens came in. That is the compression ratio:

compression = num_image_tokens / num_queries

Count the parameters with this simplified model:

• Learned queries: num_queries * d_query
• Each of num_layers cross-attention blocks has four projections, all with a bias:
  • W_Q: d_query x d_query + d_query
  • W_K: d_image x d_query + d_query
  • W_V: d_image x d_query + d_query
  • W_O: d_query x d_query + d_query
• Final projector into the LLM: d_query x d_llm + d_llm

W_K and W_V read the frozen vision encoder's width d_image (1408 for EVA-ViT-g) and write the Q-Former's own width d_query (768), which is why the two widths appear asymmetrically.

Notice what is missing from the list: num_image_tokens. Sequence length never appears in a parameter count - only in the activation/FLOP cost. The Q-Former's size is identical whether it consumes 257 or 4097 image tokens.

Your Task

Implement:

def qformer_stats(num_queries, d_query, d_image, num_image_tokens, num_layers, d_llm):

Return [num_image_tokens, num_queries, d_llm, total_params, compression] where the first four are ints and compression is num_image_tokens / num_queries rounded to 4 decimals.

Input Format

Six positive integers, as named above.

Output Format

[int, int, int, int, float]

Sample

print(qformer_stats(32, 768, 1408, 256, 12, 4096))

Output:

[256, 32, 4096, 43319296, 8.0]