PIXELBANKv8.2.1
Menu
Sign In

BPE Merge Step

EasyTokenization Algorithms

Implement a single step of the Byte Pair Encoding (BPE) merge algorithm.

Given a list of tokens (strings) and a merge pair (two consecutive tokens to merge), scan through the token list and merge every adjacent occurrence of the pair into a single token.

Example:

  • Tokens: ["l", "o", "w", "e", "r"]
  • Merge pair: ("l", "o")
  • Result: ["lo", "w", "e", "r"]

Multiple occurrences should all be merged in a single pass (left to right).

Example:

Input:
l o w e r
l o
Output:
lo w e r
Reasoning:
  • The input is split into a list of tokens: ["l", "o", "w", "e", "r"] and a merge pair: ("l", "o")
  • The algorithm scans through the token list from left to right, checking for adjacent occurrences of the merge pair ("l", "o")
  • When an occurrence of the merge pair is found, the two tokens are merged into a single token: "lo"
  • The resulting list of tokens after the merge is: ["lo", "w", "e", "r"], which is then joined into a string to produce the output: lo w e r

Constraints:

  • Input line 1: space-separated tokens
  • Input line 2: two tokens separated by space (the merge pair)
  • Output: space-separated merged tokens
  • Merge all occurrences in one left-to-right pass
Editor

Test Results

0/0
Run code to see test results.