AnyRes Tiling Token Budget

Problem Statement

LLaVA-NeXT feeds high-resolution images to a fixed-resolution vision encoder by tiling: the image is split into a grid of encoder-sized tiles, each tile is encoded separately, and a downscaled thumbnail of the whole image is encoded too. Compute the resulting visual-token budget.

Background

A CLIP/SigLIP encoder is trained at one resolution (say 336x336). Feeding it a 672x1008 document at native resolution is not possible, and downsampling the document to 336x336 destroys the text. AnyRes solves this by cutting the image into tiles:

rows = ceil(image_h / tile_size)
cols = ceil(image_w / tile_size)
num_tiles = rows * cols

Every tile is encoded independently at the encoder's native resolution, and one extra thumbnail tile (the whole image squashed to tile_size x tile_size) is encoded so the model keeps global context:

total_tiles = num_tiles + 1   (when the thumbnail is included)

Each tile produces (tile_size // patch_size)^2 patch tokens. Because that number explodes - 336/14 = 24, so 576 tokens per tile - implementations apply token compression, most simply a pool_stride x pool_stride average-pool over the patch grid before projection:

side = (tile_size // patch_size) // pool_stride
tokens_per_tile = side * side
total_tokens = total_tiles * tokens_per_tile

This is the single most important number in a VLM's context budget: at pool_stride = 1 a 2x3 tiling costs 7 * 576 = 4032 positions, which can exceed the text prompt by two orders of magnitude.

Your Task

Implement:

def anyres_token_budget(image_h, image_w, tile_size, patch_size, pool_stride=1, include_thumbnail=True):

Return [rows, cols, total_tiles, tokens_per_tile, total_tokens] as plain Python integers. Note that total_tiles already includes the thumbnail when include_thumbnail is True.

Input Format

• image_h, image_w - source image size in pixels (need NOT be a multiple of tile_size)
• tile_size - the encoder's native input resolution
• patch_size - ViT patch size; tile_size is divisible by patch_size
• pool_stride - side of the square pooling window over the patch grid; tile_size // patch_size is divisible by it
• include_thumbnail - whether the global thumbnail tile is encoded as well

Output Format

A list of five integers.

Sample

print(anyres_token_budget(672, 1008, 336, 14))

Output:

[2, 3, 7, 576, 4032]