PIXELBANKv8.2.1
Menu
Sign In

WordPiece Tokenizer

MediumTokenization Algorithms

Implement a greedy longest-match WordPiece tokenizer.

Given a vocabulary (set of subword tokens) and a word, tokenize the word using greedy longest-match from left to right. Subword tokens after the first piece are prefixed with "##".

Rules:

  • Start from the beginning of the word
  • Find the longest prefix that exists in the vocabulary
  • If found, add it as a token; continue with the remainder (prefix "##" to remaining pieces)
  • If no match is found for a character, output [UNK] for the entire word

Input:

  • Line 1: space-separated vocabulary tokens
  • Line 2: word to tokenize

Output: space-separated tokens

Example:

Input:
un ##able ##ing play ##s playing
playing
Output:
play ##ing
Reasoning:
  • The vocabulary tokens are un, ##able, ##ing, play, ##s, and playing, and the word to tokenize is playing.
  • Starting from the beginning of the word, the longest prefix that exists in the vocabulary is play.
  • The remainder of the word is ing, which is found in the vocabulary as ##ing, so it is added as the next token.
  • The final output is the combination of these two tokens: play ##ing.

Constraints:

  • Vocabulary contains lowercase subword tokens (some prefixed with ##)
  • Word is a single lowercase string with no spaces
  • Output [UNK] if any portion can't be matched
Editor

Test Results

0/0
Run code to see test results.