words

Definition

Words as the basic units of language — though modern models rarely operate on them directly, splitting text into subword tokens instead, so a single word may span several tokens. The term persists in titles largely through the embedding literature, where mapping words to vectors was the central object of study. When precision matters, papers say tokens, not words.

Explainers using this term

• Build Your Own BPE Tokenizer — Learning Merge Rules, and Getting Punished by JapaneseNeural Machine Translation of Rare Words with Subword Units
• Tokenizers from Scratch — The Unit an LLM Cuts the World IntoNeural Machine Translation of Rare Words with Subword Units
• Paper Deep Dive — ViT: Treating an Image Like a SentenceAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

In the textbook

• ViT — Treating Images Like Words
• CLIP — Words and Images on One Map