Tokenizers from Scratch — The Unit an LLM Cuts the World Into

An LLM reads neither characters nor words. How BPE builds a vocabulary, what SentencePiece actually fixed, why some languages pay more for the same sentence, and what you trade away when you grow the vocabulary — worked by hand and in code, from zero.

Neural Machine Translation of Rare Words with Subword Units

An LLM never sees characters

When you send a model a sentence, what enters it is not text. It is a sequence of integers. The thing that turns a string into those integers is the tokenizer, and the unit it works in is the token.

Deciding how to number things — where to cut the text, what counts as one piece — is not done by the neural network. It is done by a separate program that is frozen before training begins. And that choice propagates into your bill, your context limit, your compute cost, and how well the model handles one language versus another. Tokenization looks like boring preprocessing. It is really the resolution at which the model sees the world.

An analogy: designing a box of LEGO

Think of writing as building with LEGO, where you get to choose which brick shapes go in the box — say, fifty thousand of them.

• Stock only 1×1 bricks. You can build anything, but a castle takes tens of thousands of pieces.
• Stock finished models: "castle," "house," "ship." One piece and you're done — but a "lighthouse" that isn't in the box is simply impossible.

The first is a character-level tokenizer, the second a word-level one. Characters give you a vocabulary of a few hundred and zero unknowns, at the price of brutally long sequences. Words give you tokens that carry meaning, at the price of a vocabulary in the hundreds of thousands — and even then new coinages, proper nouns and typos fall through the cracks into the UNK bin.

What everyone actually ships is the middle design: subwords. Frequent words get their own piece; rare ones are spelled out from fragments. You don't need "lighthouse" in the box if you have "light" and "house."

How BPE works: glue the most common neighbors

The dominant way to build a subword vocabulary is BPE (Byte Pair Encoding). It began life in 1994 as a data compression algorithm and was brought to machine translation in 2016. The compression heritage is not a coincidence — "give frequent patterns a short code" is the same instinct behind entropy coding.

The procedure is almost embarrassingly simple.

1. Split everything into single characters (that's your starting vocabulary)
2. Across the whole corpus, count which adjacent pair occurs most often
3. Add that pair to the vocabulary as one new token and glue every occurrence together
4. Repeat 2–3 until the vocabulary reaches its target size

Equation (1) says only this: of everything you just counted, take the adjacent pair $(a,b)$ that appeared most. $\arg\max$ means "return the argument that maximizes this," and $\mathrm{count}(a,b)$ is the number of times $b$ directly followed $a$.

Read that in words: keep a tally for every neighboring pair, then point at the tallest column. The star on $(a,b)^{*}$ just marks the winner of that round.

Work it by hand. Say the corpus is low ×5, lower ×2, newest ×6. As characters: l o w / l o w e r / n e w e s t. The most frequent adjacent pair is e s (6), so es is born; next comes es t giving est — and so on, growing outward from the densest clusters. Fragments that look meaningful to a human, like low and est, fall out on their own. But they are a byproduct of statistics: no grammar and no dictionary were involved.

def merge_once(corpus):                       # corpus: list[list[str]]
    pairs = {}
    for toks in corpus:
        for p in zip(toks, toks[1:]):
            pairs[p] = pairs.get(p, 0) + 1
    a, b = max(pairs, key=pairs.get)           # exactly equation (1)
    return a + b, [_apply(t, a, b, a + b) for t in corpus]

Training is just this loop run tens of thousands of times. Inference is just replaying the learned merge rules in the order they were learned. The order is the rule — swap two lines of the merge table and you have a different tokenizer.

Byte-level BPE: abolishing the unknown character

Starting from characters means any character absent from the training data — an emoji, a rare kanji, a Cyrillic letter — is unknown forever. GPT-2's answer was byte-level BPE: start from UTF-8 bytes instead of characters. There are only 256 of them, and every possible text is a byte string, so unknown tokens stop existing by construction.

Note one consequence, because it comes back later. In UTF-8, ASCII letters and digits are one byte each, but most CJK characters — kana, kanji, Chinese, Korean syllables — take three. If no merge rule happens to cover them, a single character can cost up to three tokens.

WordPiece and Unigram: choosing by probability, not frequency

BPE has two siblings. WordPiece (used by BERT and friends) picks its merges not by raw count but by a score along the lines of $\mathrm{count}(ab)/(\mathrm{count}(a)\cdot\mathrm{count}(b))$. The difference: it prefers pairs that appear together more than they appear apart, rather than pairs that are merely common.

The unigram language model (one of SentencePiece's default modes) runs the other way around. It starts from an oversized candidate vocabulary and prunes what it doesn't need. Each token carries a probability $p(x)$, and a segmentation is scored by its likelihood:

Equation (2) says: how plausible a segmentation is equals the probabilities of every token you used, multiplied together. $\prod$ is the instruction "multiply all of these together," $\mathbf{x}$ is the segmentation (a sequence of tokens), and $p(x_i)$ is how likely the $i$-th token is.

Or in words: a cut scores well when it is built out of common pieces, and few of them. Slip in one rare piece and you multiply by a small number, which drags the whole score down — so the winning split is the one that spells the string using ordinary parts. Since one string can be cut many different ways, you take the cut that maximizes this. The number of candidates is exponential, but dynamic programming — filling in "best segmentation up to this position" left to right — solves it efficiently.

The payoff is that the segmentation is not forced to be unique. Because probabilities are attached, training can deliberately sample the second- or third-most-likely cut (subword regularization), which stops the model from over-relying on one particular way of splitting a word.

SentencePiece adds one more idea on top: assume nothing about the language. Older tokenizers quietly assumed a first step of "split on whitespace." For Japanese, Chinese and Thai, which don't put spaces between words, that assumption collapses immediately. SentencePiece consumes the raw string and treats the space itself as an ordinary character, ▁ (U+2581), that lives in the vocabulary. Concatenate the pieces, turn ▁ back into a space, and you recover the original string exactly — lossless round-tripping is its headline property.

Why some languages pay more

With all that in place, the disadvantage stacks up in three layers.

1. The bytes are heavy. In UTF-8, kana and kanji are typically three bytes. If no merge covering that character made it into the vocabulary, one character inflates to two or three tokens.

2. The vocabulary mirrors the training data. BPE and unigram alike adopt "sequences that show up a lot in the corpus." If the corpus is English-heavy, English suffixes like ing and tion get dedicated tokens while common Japanese expressions don't.

3. There is no whitespace. In implementations whose pre-tokenizer splits on spaces, a Japanese sentence looks like one enormous word. SentencePiece solved this, but tokenizers that pre-split with English-oriented regexes still lose accuracy here.

The result is that the same content, written in Japanese, tends to cost more tokens than in English. That is not merely an aesthetic complaint: APIs bill per token, context limits are counted in tokens, and the cost of self-attention grows with the square of sequence length.

So poor token efficiency hits price, context and speed at once. When a model claims it "retrained its tokenizer" for a language, that is a statement about accuracy and equally a statement about unit cost.

The vocabulary-size trade

Should we just make the vocabulary enormous, then? This is the most interesting trade in the whole design.

Growing $|V|$ means each token covers more characters, so sequences get shorter. Shorter sequences cut attention cost quadratically, fit more content into the same context window, and shrink the KV cache that inference accumulates.

Three things are paid in exchange. The two ends get fat: the embedding matrix is $|V| \times d$ (with $d$ the model dimension), and the output softmax has the same shape — the smaller the model, the larger the fraction those ends eat. Rare tokens never learn: the tail of a big vocabulary is full of tokens that almost never appear, and their embeddings receive almost no training signal. The output layer costs more: every generated token requires scoring and normalizing over the entire vocabulary.

Which is why real models size their vocabulary against their target language mix and their own parameter count. GPT-2 uses 50,257; Llama 2, 32,000; Llama 3, 128,256; Gemma, 256,000. Bigger is not newer-and-better — it means the model targets more languages, or is large enough to absorb the cost of a large vocabulary.

Getting your hands on it

Measuring how many tokens your own text costs takes a few lines.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")

s = "Tokenization is not preprocessing. It is design."
ids = tok.encode(s)
print(len(s), len(ids))                 # characters vs tokens
print(tok.convert_ids_to_tokens(ids))   # look at where it actually cut

The trick is to look at the cut points, not just the count. A number alone won't tell you that your product names are being shredded one character at a time. If you're training your own vocabulary, SentencePiece is short:

import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input="corpus.txt", model_prefix="ja", vocab_size=32000,
    model_type="unigram", character_coverage=0.9995, byte_fallback=True)

character_coverage is the fraction of distinct characters the vocabulary covers directly; for character-rich languages like Japanese and Chinese, roughly 0.9995 is recommended rather than 1.0. byte_fallback is what catches the remainder instead of dumping it into UNK.

How this shows up on the job

Who touches it, and when. Application engineers estimating cost and context budgets; fine-tuning engineers deciding whether a domain vocabulary is worth adding; anyone on multilingual support who has to explain why this language is slower and pricier.

Tools and parameters you'll actually type. On the OpenAI side, tiktoken and encoding names like cl100k_base. For open models, HuggingFace AutoTokenizer / tokenizers; for training, sentencepiece (--vocab_size, --model_type, --character_coverage, --byte_fallback). Adding vocabulary always pairs tokenizer.add_tokens() with model.resize_token_embeddings().

Pitfalls that turn into incidents.

• Estimating in characters. Billing, max_tokens and context limits are all token-based, and the ratio to characters varies wildly by language. Measure before you design.
• Trailing whitespace. In most BPE vocabularies the space attaches to the front of the next word. Ending a prompt with a space blocks those "space + word" tokens from being used and can make generations subtly worse.
• Stop strings that don't land on token boundaries. Your stop sequence is a string; the model emits tokens. Misalignment means it either doesn't stop or overshoots.
• Tokenizer/checkpoint mismatch. Add tokens but forget to resize the embedding matrix and you get out-of-range IDs — or a silent misalignment. Newly added embeddings are randomly initialized, so they mean nothing until trained.
• Digits. How a run of digits is chunked is tokenizer-specific, and inconsistent grouping makes arithmetic unstable. Route real computation to a tool call.
• Normalization erases information. SentencePiece's NFKC-style normalization folds full-width and half-width forms and some punctuation. If those distinctions carry meaning in your data, check the setting.

The question you'll get asked. "Why does the same prompt cost more in Japanese?" The chain of reasoning is four links: UTF-8 byte length → corpus skew shaping which merges exist → sequence length → quadratic attention cost and per-token billing. If the follow-up is "so just enlarge the vocabulary," answer with the fat ends, the untrained rare tokens, and the need to balance against model size.

Summary

• The tokenizer is a separate program, frozen before training, that sets the resolution at which the model sees the world
• BPE is nothing but "glue the most frequent adjacent pair," repeated; WordPiece picks by a score, unigram by likelihood
• SentencePiece dropped the whitespace assumption and made the process lossless, which brought space-free languages onto equal footing
• The disadvantage for Japanese stacks three ways — three-byte characters, English-skewed corpora, no whitespace — and it lands on price and speed
• Vocabulary size trades shorter sequences against fatter ends and undertrained rare tokens

Next comes the step that turns those tokens into vectors: Embeddings from Scratch.