Loading...
Build a character-level tokenizer that converts text to token IDs and back.
Given a text string, build a vocabulary mapping each unique character to an integer ID (sorted by character order, starting from 0). Then encode the text as a list of IDs and decode it back.
Input:
Output:
hello
1 0 2 2 3 hello
h maps to 0, e maps to 1, l maps to 2, o maps to 3.h (0) is not present, so we start with h's ID which is not in the list, e's ID is 1, l's ID is 2, the next l's ID is also 2, and o's ID is 3. However, the correct mapping should be based on the sorted character order, so e maps to 0, h maps to 1, l maps to 2, o maps to 3. Thus, "hello" becomes 1 0 2 2 3.h, 0 maps to e, 2 maps to l, 2 maps to l, and 3 maps to o, resulting in "hello".