BPE Tokenizer Explained
"TLDR: This article introduces the application of the Byte Pair Encoding (BPE) algorithm in tokenization scenarios. Traditional text tokenization methods include word-based and character-based approaches, while for Chinese, there is also a common-word-based tokenization method. In the era of large language models (LLMs), the BPE algorithm has been adopted due to its suitability for massive datasets. By comparing tokenization methods at different levels, such as char level, word level, and subword level, the article highlights their respective advantages, disadvantages, and applicable scenarios."
Traditional text tokenization methods are basically based on two approaches: character-level and word-level. For Chinese, there are also common-word-based tokenization methods (such as jieba). In the era of LLMs, the Byte Pair Encoding algorithm is more suitable for tokenizing massive amounts of data.
Background
For example, consider the sentence "i am highest."
-
Character-level tokenization:
['i', 'a', 'm', ' ', 'h', 'i', 'g', 'h', 'e', 's', 't']. This level of tokenization produces very long token sequences for a single sentence. For longer sentences, the token length becomes even more explosive, making training costs too high. -
Word-level tokenization:
['i', 'am', 'highest']. This looks fine at first glance. However, when encountering words like high/higher/highest, this approach treats them as completely different words, which isn't ideal since they essentially share the same meaning. Additionally, when the vocabulary isn't rich enough, it easily leads to OOV (out-of-vocabulary) problems. -
Subword-level tokenization: This is a compromise between the two approaches above, splitting a word into multiple subword representations. "highest" would be split into
['high', 'est'].
Principle
Byte Pair Encoding is one specific implementation of subword-level tokenization. BPE obtains tokenization results by iteratively mining frequent character pairs and replacing them. It first splits words into individual characters, then repeatedly replaces the most frequent pair of characters with another character, until the iteration count is reached. It's somewhat similar to Huffman coding, compressing common patterns.
It sounds a bit convoluted, but the process becomes clear when you see it:
corpus = ["highest", "higher", "lower", "lowest", "cooler", "coolest"]
{
"highest": ["h", "i", "g", "h", "e", "s", "t", "</w>"],
"higher": ["h", "i", "g", "h", "e", "r", "</w>"],
"lower": ["l", "o", "w", "e", "r", "</w>"],
"lowest": ["l", "o", "w", "e", "s", "t", "</w>"],
"cooler": ["c", "o", "o", "l", "e", "r", "</w>"],
"collest": ["c", "o", "o", "l", "e", "s", "t", "</w>"],
}
{
"highest": ["h", "i", "g", "h", "es", "t", "</w>"],
"higher": ["h", "i", "g", "h", "e", "r", "</w>"],
"lower": ["l", "o", "w", "e", "r", "</w>"],
"lowest": ["l", "o", "w", "es", "t", "</w>"],
"cooler": ["c", "o", "o", "l", "e", "r", "</w>"],
"collest": ["c", "o", "o", "l", "es", "t", "</w>"],
}
{
"highest": ["h", "i", "g", "h", "est", "</w>"],
"higher": ["h", "i", "g", "h", "e", "r", "</w>"],
"lower": ["l", "o", "w", "e", "r", "</w>"],
"lowest": ["l", "o", "w", "est", "</w>"],
"cooler": ["c", "o", "o", "l", "e", "r", "</w>"],
"collest": ["c", "o", "o", "l", "est", "</w>"],
}
{
"highest": ["h", "i", "g", "h", "est</w>"],
"higher": ["h", "i", "g", "h", "e", "r", "</w>"],
"lower": ["l", "o", "w", "e", "r", "</w>"],
"lowest": ["l", "o", "w", "est</w>"],
"cooler": ["c", "o", "o", "l", "e", "r", "</w>"],
"collest": ["c", "o", "o", "l", "est</w>"],
}
{
"highest": ["h", "i", "g", "h", "est</w>"],
"higher": ["h", "i", "g", "h", "er", "</w>"],
"lower": ["l", "o", "w", "er", "</w>"],
"lowest": ["l", "o", "w", "est</w>"],
"cooler": ["c", "o", "o", "l", "er", "</w>"],
"collest": ["c", "o", "o", "l", "est</w>"],
}
{
"highest": ["h", "i", "g", "h", "est</w>"],
"higher": ["h", "i", "g", "h", "er</w>"],
"lower": ["l", "o", "w", "er</w>"],
"lowest": ["l", "o", "w", "est</w>"],
"cooler": ["c", "o", "o", "l", "er</w>"],
"collest": ["c", "o", "o", "l", "est</w>"],
}
...
Code Implementation
The favorite part for library users:
import sentencepiece as sp
sp.SentencePieceTrainer.train(input="tlbb.txt", # Supports txt and tsv formats
model_prefix="tlbb", # Prefix for saving the model file name
vocab_size=5000,
model_type='bpe')
Handling Chinese Text
The first step of BPE is character-level tokenization. In English scenarios, this involves 26 letters plus some special characters, which is manageable. In Chinese scenarios, however, there are thousands of individual Chinese characters, making BPE less suitable. BPE splits text into minimal units based on ascii, so a single Chinese character can end up being split into garbled fragments.