Definition
Byte-pair encoding (BPE) starts with the 256 bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary has the desired size (e.g. 128,256 for Llama 3, 199,998 for GPT-4o).
Consequences:
- Strawberry problem: the model sees [str][aw][berry], never letters, so it fails at counting letters, anagrams, spelling backwards.
- Numbers are split inconsistently ([1234][5] or [12][345]).
- Low-resource languages need more tokens per word (5 to 10× more in Thai than in English).
- Glitch tokens: tokens frequent in the tokenizer data but rare in model training get almost untrained embeddings and cause erratic behavior.