#bpe

2 articles

01 ·How Transformers Work·★ MEMBER·PAPER·11 min read Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by Japanese Write the BPE trainer and encoder yourself. Why the artifact of training is an ordered rulebook rather than a vocabulary, how to stop recounting the corpus on every merge, why the first few thousand merge slots in Japanese are spent assembling characters, and how to run a vocabulary-size sweep that actually means something. 02 ·How Transformers Work·FREE·PAPER·10 min read Tokenizers from Scratch — The Unit an LLM Cuts the World Into An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero.