📘

BPE Merge Step

Implement a single step of the Byte Pair Encoding (BPE) merge algorithm.

Given a list of tokens (strings) and a merge pair (two consecutive tokens to merge), scan through the token list and merge every adjacent occurrence of the pair into a single token.

Example:

Tokens: ["l", "o", "w", "e", "r"]
Merge pair: ("l", "o")
Result: ["lo", "w", "e", "r"]

Multiple occurrences should all be merged in a single pass (left to right).

Example:

Input:

l o w e r
l o

Output:

lo w e r

Reasoning:

The input is split into a list of tokens: ["l", "o", "w", "e", "r"] and a merge pair: ("l", "o")
The algorithm scans through the token list from left to right, checking for adjacent occurrences of the merge pair ("l", "o")
When an occurrence of the merge pair is found, the two tokens are merged into a single token: "lo"
The resulting list of tokens after the merge is: ["lo", "w", "e", "r"], which is then joined into a string to produce the output: lo w e r

Constraints:

Input line 1: space-separated tokens
Input line 2: two tokens separated by space (the merge pair)
Output: space-separated merged tokens
Merge all occurrences in one left-to-right pass

Editor

Python 3.13.1

Test Results

0/0

Run code to see test results.