CLS and Register Token Sequence Length

1. Background Knowledge

In a Vision Transformer (ViT), an input image is split into a grid of non-overlapping patches. If the grid has rows and cols patches, the total number of patch tokens is simply the product of these two dimensions. These patch embeddings form the core of the sequence that the transformer processes.

To enable classification tasks, a learnable [CLS] (class) token is typically prepended to the sequence. This single token aggregates information from all other tokens through self-attention and is used for the final classification head. In more recent architectures like DINOv2, register tokens are introduced. These are additional learnable tokens that act as "sink" tokens, absorbing high-norm activations that can otherwise disrupt the attention patterns of the actual patch tokens. While they carry no direct image content, they occupy positions in the sequence and participate in the attention computation, thereby increasing the computational cost.

The total sequence length $L$ is the sum of the patch tokens, the CLS token (if present), and the register tokens. Mathematically, this is expressed as:

where $\mathbb{1}[\text{use\_cls}]$ is an indicator function that equals 1 if the CLS token is used, and 0 otherwise.

2. Algorithm Approach

This is a straightforward arithmetic problem. The approach involves:

1. Calculating the number of patch tokens by multiplying rows and cols.
2. Conditionally adding 1 to the total if use_cls is True.
3. Adding the value of n_register to the total.
4. Returning the final sum as an integer.