Interleaved Multimodal Sequence Layout

mediumPROPatch Embedding & Visual TokensVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

A VLM prompt is a single flat token sequence in which visual tokens are spliced between text tokens. Given the segment layout, compute the total sequence length, the index span each image occupies, and the fraction of the context spent on pixels.

Background

After the bridge module produces n visual tokens for an image, they are inserted into the language model's token stream exactly where the <image> placeholder appeared. Nearly every implementation wraps each block in delimiter tokens (<img_start> / <img_end>) so the decoder can tell text from vision:

... text ... <img_start> v1 v2 ... vn <img_end> ... text ...

With delimiters an image therefore costs n + 2 positions, of which only the inner n are visual embeddings; the two delimiters are ordinary learned text-vocabulary embeddings. For an interleaved multi-image prompt the spans simply accumulate left to right.

Knowing the spans is not bookkeeping trivia - it is what SFT loss masking, attention analysis and visual-token pruning all index into. And the visual_fraction is the number that tells you a 576-token image next to a 12-token question means 96%+ of the forward pass is spent on the picture.

Your Task

Implement:

def build_sequence(segments, use_delimiters=True):

segments is a list of (kind, count) pairs, in order, where kind is either "text" or "image". A "text" segment contributes count positions. An "image" segment contributes count visual positions, wrapped in one opening and one closing delimiter position when use_delimiters is True.

Return [total_len, spans, visual_fraction] where:

total_len is an int, the length of the whole sequence
spans is a list of [start, end] pairs, one per image, giving the half-open index range of the visual tokens only (delimiters excluded), 0-indexed
visual_fraction is visual_tokens / total_len rounded to 4 decimals

Input Format

A list of (str, int) tuples and a boolean.

Output Format

[int, [[int, int], ...], float]

Sample

print(build_sequence([("text", 12), ("image", 576), ("text", 8)]))

Output:

[598, [[13, 589]], 0.9632]

Example

Input

print(build_sequence([("text", 12), ("image", 576), ("text", 8)]))

Output

[598, [[13, 589]], 0.9632]

Reasoning

12 text tokens occupy 0..11. The opening delimiter takes index 12, so the 576 visual tokens run from 13 to 588, i.e. the half-open span [13, 589). The closing delimiter is 589, then 8 text tokens: 12 + 1 + 576 + 1 + 8 = 598. 576/598 = 0.96321... -> 0.9632.

Constraints

kind is exactly "text" or "image"
All counts are non-negative integers; segments may be empty
Spans are half-open [start, end) and cover ONLY the visual tokens, not the delimiters
visual_fraction must be rounded to 4 decimals; return 0.0 when the sequence is empty
Positions are 0-indexed

Learn this in

Advanced NLP TopicsNLP Study Plan Introduction to Vision-Language ModelsVLM Study Plan Language Model BackboneVLM Study Plan Instruction Tuning & the LLaVA ParadigmVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

In Vision-Language Models (VLMs), the input to the transformer is a single, unified sequence of tokens. Unlike traditional NLP models that only process text, VLMs must interleave visual information with linguistic context. This is achieved by converting images into a sequence of visual tokens (often via a vision encoder and a projection layer) and inserting them into the text stream at specific placeholder positions (e.g., <image>).

A critical implementation detail is the use of delimiter tokens. To help the language model distinguish between textual semantics and visual embeddings, implementations often wrap visual token blocks with special tokens like <img_start> and <img_end>. These delimiters are part of the text vocabulary and consume positions in the sequence, but they are not visual embeddings. Therefore, when calculating metrics like "visual fraction" or defining attention masks, one must carefully separate the structural tokens (delimiters) from the content tokens (visual embeddings).

Understanding half-open intervals is essential for indexing. In Python and many ML frameworks, a span $[start, end)$ includes the index start but excludes end. This convention simplifies length calculations ( $length = end - start$ ) and slicing operations. When processing interleaved sequences, maintaining a running cursor (current index) allows you to map each segment to its precise location in the final flat array.

2. Algorithm Approach

The problem requires simulating the construction of a flat token sequence from a list of segments. The core algorithmic pattern is linear iteration with state accumulation. You will iterate through the segments list once, maintaining a current_index variable that tracks the position in the hypothetical output sequence.

For each segment, you determine its contribution to the total length and update the current_index. If the segment is an image, you also record the span of the visual tokens specifically. The logic branches based on the kind of the segment:

Text Segments: Simply advance the index by the count.
Image Segments: Advance the index by the visual count plus the delimiter overhead (if enabled). Record the start and end indices of the visual portion only.

Finally, compute the aggregate statistics: total length, the list of recorded spans, and the ratio of visual tokens to the total length.

3. Step-by-Step Strategy

Initialize Variables:

Create an empty list spans to store [start, end] pairs for images.
Initialize current_index = 0.
Initialize total_visual_tokens = 0.

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results