rare

Definition

Low-frequency items — words, entities, or situations the model saw few examples of. Rare words were the specific problem subword tokenization was invented to solve: a fixed word vocabulary drops them as unknown, while splitting into pieces keeps them representable. More generally, behavior on rare cases is invisible in aggregate accuracy, so it has to be measured separately whenever it matters.

Explainers using this term

• Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by JapaneseNeural Machine Translation of Rare Words with Subword Units
• Tokenizers from Scratch — The Unit an LLM Cuts the World IntoNeural Machine Translation of Rare Words with Subword Units