tokenizer

Definition

The component that cuts raw text into the discrete units a model consumes, using a vocabulary and a segmentation scheme — BPE merges, unigram scoring, or WordPiece, depending on the method — fixed at training time and frozen thereafter. Though it sits outside the network, it has real consequences: a poor fit inflates the token count for a given text, which directly costs money and context budget, and it shapes how well the model handles numbers, code, and under-represented languages.