Interleaved Multimodal Sequence Layout
Problem Statement
A VLM prompt is a single flat token sequence in which visual tokens are spliced between text tokens. Given the segment layout, compute the total sequence length, the index span each image occupies, and the fraction of the context spent on pixels.
Background
After the bridge module produces n visual tokens for an image, they are inserted into the language model's token stream exactly where the <image> placeholder appeared. Nearly every implementation wraps each block in delimiter tokens (<img_start> / <img_end>) so the decoder can tell text from vision:
... text ... <img_start> v1 v2 ... vn <img_end> ... text ...
With delimiters an image therefore costs n + 2 positions, of which only the inner n are visual embeddings; the two delimiters are ordinary learned text-vocabulary embeddings. For an interleaved multi-image prompt the spans simply accumulate left to right.
Knowing the spans is not bookkeeping trivia - it is what SFT loss masking, attention analysis and visual-token pruning all index into. And the visual_fraction is the number that tells you a 576-token image next to a 12-token question means 96%+ of the forward pass is spent on the picture.
Your Task
Implement:
def build_sequence(segments, use_delimiters=True):
segments is a list of (kind, count) pairs, in order, where kind is either "text" or "image". A "text" segment contributes count positions. An "image" segment contributes count visual positions, wrapped in one opening and one closing delimiter position when use_delimiters is True.
Return [total_len, spans, visual_fraction] where:
- total_len is an int, the length of the whole sequence
- spans is a list of [start, end] pairs, one per image, giving the half-open index range of the visual tokens only (delimiters excluded), 0-indexed
- visual_fraction is visual_tokens / total_len rounded to 4 decimals
Input Format
A list of (str, int) tuples and a boolean.
Output Format
[int, [[int, int], ...], float]
Sample
print(build_sequence([("text", 12), ("image", 576), ("text", 8)]))
Output:
[598, [[13, 589]], 0.9632]
Example:
print(build_sequence([("text", 12), ("image", 576), ("text", 8)]))[598, [[13, 589]], 0.9632]
12 text tokens occupy 0..11. The opening delimiter takes index 12, so the 576 visual tokens run from 13 to 588, i.e. the half-open span [13, 589). The closing delimiter is 589, then 8 text tokens: 12 + 1 + 576 + 1 + 8 = 598. 576/598 = 0.96321... -> 0.9632.
Constraints:
kindis exactly"text"or"image"- All counts are non-negative integers;
segmentsmay be empty - Spans are half-open
[start, end)and cover ONLY the visual tokens, not the delimiters visual_fractionmust be rounded to 4 decimals; return0.0when the sequence is empty- Positions are 0-indexed
1. Background Knowledge
In Vision-Language Models (VLMs), the input to the transformer is a single, unified sequence of tokens. Unlike traditional NLP models that only process text, VLMs must interleave visual information with linguistic context. This is achieved by converting images into a sequence of visual tokens (often via a vision encoder and a projection layer) and inserting them into the text stream at specific placeholder positions (e.g., <image>).
A critical implementation detail is the use of delimiter tokens. To help the language model distinguish between textual semantics and visual embeddings, implementations often wrap visual token blocks with special tokens like <img_start> and <img_end>. These delimiters are part of the text vocabulary and consume positions in the sequence, but they are not visual embeddings. Therefore, when calculating metrics like "visual fraction" or defining attention masks, one must carefully separate the structural tokens (delimiters) from the content tokens (visual embeddings).
Understanding half-open intervals is essential for indexing. In Python and many ML frameworks, a span [start,end) includes the index start but excludes end. This convention simplifies length calculations (length=end−start) and slicing operations. When processing interleaved sequences, maintaining a running cursor (current index) allows you to map each segment to its precise location in the final flat array.
2. Algorithm Approach
The problem requires simulating the construction of a flat token sequence from a list of segments. The core algorithmic pattern is linear iteration with state accumulation. You will iterate through the segments list once, maintaining a current_index variable that tracks the position in the hypothetical output sequence.
For each segment, you determine its contribution to the total length and update the current_index. If the segment is an image, you also record the span of the visual tokens specifically. The logic branches based on the kind of the segment:
- Text Segments: Simply advance the index by the count.
- Image Segments: Advance the index by the visual count plus the delimiter overhead (if enabled). Record the start and end indices of the visual portion only.
Finally, compute the aggregate statistics: total length, the list of recorded spans, and the ratio of visual tokens to the total length.
3. Step-by-Step Strategy
- Initialize Variables:
- Create an empty list spans to store [start, end] pairs for images.
- Initialize current_index = 0.
- Initialize total_visual_tokens = 0.
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.