#tokenizer
3 articles
01
·How Transformers Work·★ MEMBER·PAPER·11 min read
Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by Japanese
Write the BPE trainer and encoder yourself. Why the artifact of training is an ordered rulebook rather than a vocabulary, how to stop recounting the corpus on every merge, why the first few thousand merge slots in Japanese are spent assembling characters, and how to run a vocabulary-size sweep that actually means something.
02
·How Transformers Work·FREE·PAPER·10 min read
Tokenizers from Scratch — The Unit an LLM Cuts the World Into
An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero.
03
·CNNs & Image Recognition·★ MEMBER·PAPER·10 min read
Paper Walkthrough: PhiZero — A World Model That Reasons in a Language of Physics Before It Renders
A close read of PhiZero, which reasons about how the world will change in a discrete 'physical language' learned self-supervised from video, and only then renders the footage — instead of predicting pixels directly. The two-stage tokenizer and reasoner, the top spot on Physics-IQ Verified, zero-shot motion transfer, and the limitations the paper itself admits.