sentencepiece

Definition

An open-source tokenizer from Google, notable for being language-independent: it trains directly on raw text without assuming whitespace-separated words, which lets Japanese, Chinese, and Thai go through the same pipeline as English. Spaces are encoded as an ordinary symbol, so detokenization is deterministic rather than rule-driven — though any normalization applied first means the recovered string need not match the original byte for byte. It implements both BPE and unigram segmentation.