subword

Definition

A unit between a character and a word. Frequent words stay whole while rare ones are split into pieces, so a fixed-size vocabulary can represent any string with no unknown-word fallback. It is the practical compromise between word-level tokenization, which breaks on anything unseen, and character-level, which lengthens sequences. Most widely used language models segment this way, though byte-level and tokenizer-free alternatives are an active line of work.

Explainers using this term

• Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by JapaneseNeural Machine Translation of Rare Words with Subword Units
• Tokenizers from Scratch — The Unit an LLM Cuts the World IntoNeural Machine Translation of Rare Words with Subword Units