Interleaved Multimodal Sequence Layout

Problem Statement

A VLM prompt is a single flat token sequence in which visual tokens are spliced between text tokens. Given the segment layout, compute the total sequence length, the index span each image occupies, and the fraction of the context spent on pixels.

Background

After the bridge module produces n visual tokens for an image, they are inserted into the language model's token stream exactly where the <image> placeholder appeared. Nearly every implementation wraps each block in delimiter tokens (<img_start> / <img_end>) so the decoder can tell text from vision:

... text ... <img_start> v1 v2 ... vn <img_end> ... text ...

With delimiters an image therefore costs n + 2 positions, of which only the inner n are visual embeddings; the two delimiters are ordinary learned text-vocabulary embeddings. For an interleaved multi-image prompt the spans simply accumulate left to right.

Knowing the spans is not bookkeeping trivia - it is what SFT loss masking, attention analysis and visual-token pruning all index into. And the visual_fraction is the number that tells you a 576-token image next to a 12-token question means 96%+ of the forward pass is spent on the picture.

Your Task

Implement:

def build_sequence(segments, use_delimiters=True):

segments is a list of (kind, count) pairs, in order, where kind is either "text" or "image". A "text" segment contributes count positions. An "image" segment contributes count visual positions, wrapped in one opening and one closing delimiter position when use_delimiters is True.

Return [total_len, spans, visual_fraction] where:

• total_len is an int, the length of the whole sequence
• spans is a list of [start, end] pairs, one per image, giving the half-open index range of the visual tokens only (delimiters excluded), 0-indexed
• visual_fraction is visual_tokens / total_len rounded to 4 decimals

Input Format

A list of (str, int) tuples and a boolean.

Output Format

[int, [[int, int], ...], float]

Sample

print(build_sequence([("text", 12), ("image", 576), ("text", 8)]))

Output:

[598, [[13, 589]], 0.9632]