#sentencepiece

1 articles

01 ·How Transformers Work·FREE·PAPER·10 min read Tokenizers from Scratch — The Unit an LLM Cuts the World Into An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero.