CLS and Register Token Sequence Length
Problem Statement
A ViT prepends a learnable [CLS] token and, in newer designs (DINOv2, "registers"), a few extra register tokens to the patch sequence before the transformer. Compute the final sequence length fed to the encoder.
Background
The transformer input length is
L=Rpatches​⋅Cpatches​+[CLS]+nregister​
The CLS token contributes exactly one slot when present; register tokens add n_register more. These extra tokens carry no image content but occupy real positions in the attention matrix, so they count toward compute.
Your Task
Implement:
def sequence_length(rows, cols, use_cls=True, n_register=0):
Return the total sequence length as an int.
Input Format
- rows, cols (int): patch grid dimensions.
- use_cls (bool): whether a CLS token is prepended.
- n_register (int): number of register tokens.
Output Format
- A single int.
Sample
print(sequence_length(14, 14, True, 0))
Output:
197
Example:
print(sequence_length(14, 14, True, 0))
197
- Calculate the number of image patch tokens by multiplying the grid dimensions: 14×14=196.
- Determine the contribution of the CLS token; since
use_clsis True, it adds exactly 1 slot to the sequence. - Add the number of register tokens; since
n_registeris 0, this adds 0 to the total. - Sum these components to find the total sequence length: 196+1+0=197.
- The final output is 197
Constraints:
1 <= rows, cols <= 256,0 <= n_register <= 64- Add 1 only when
use_clsis true. - Return an int.
1. Background Knowledge
In a Vision Transformer (ViT), an input image is split into a grid of non-overlapping patches. If the grid has rows and cols patches, the total number of patch tokens is simply the product of these two dimensions. These patch embeddings form the core of the sequence that the transformer processes.
To enable classification tasks, a learnable [CLS] (class) token is typically prepended to the sequence. This single token aggregates information from all other tokens through self-attention and is used for the final classification head. In more recent architectures like DINOv2, register tokens are introduced. These are additional learnable tokens that act as "sink" tokens, absorbing high-norm activations that can otherwise disrupt the attention patterns of the actual patch tokens. While they carry no direct image content, they occupy positions in the sequence and participate in the attention computation, thereby increasing the computational cost.
The total sequence length L is the sum of the patch tokens, the CLS token (if present), and the register tokens. Mathematically, this is expressed as:
L=(rows×cols)+1[use_cls]+nregister​where 1[use_cls] is an indicator function that equals 1 if the CLS token is used, and 0 otherwise.
2. Algorithm Approach
This is a straightforward arithmetic problem. The approach involves:
- Calculating the number of patch tokens by multiplying rows and cols.
- Conditionally adding 1 to the total if use_cls is True.
- Adding the value of n_register to the total.
- Returning the final sum as an integer.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.