📘

WordPiece Tokenizer

Implement a greedy longest-match WordPiece tokenizer.

Given a vocabulary (set of subword tokens) and a word, tokenize the word using greedy longest-match from left to right. Subword tokens after the first piece are prefixed with "##".

Rules:

Start from the beginning of the word
Find the longest prefix that exists in the vocabulary
If found, add it as a token; continue with the remainder (prefix "##" to remaining pieces)
If no match is found for a character, output [UNK] for the entire word

Input:

Line 1: space-separated vocabulary tokens
Line 2: word to tokenize

Output: space-separated tokens

Example:

Input:

un ##able ##ing play ##s playing
playing

Output:

play ##ing

Reasoning:

The vocabulary tokens are un, ##able, ##ing, play, ##s, and playing, and the word to tokenize is playing.
Starting from the beginning of the word, the longest prefix that exists in the vocabulary is play.
The remainder of the word is ing, which is found in the vocabulary as ##ing, so it is added as the next token.
The final output is the combination of these two tokens: play ##ing.

Constraints:

Vocabulary contains lowercase subword tokens (some prefixed with ##)
Word is a single lowercase string with no spaces
Output [UNK] if any portion can't be matched

Editor

Python 3.13.1

Test Results

0/0

Run code to see test results.