detokenizer

Definition

The inverse step: turning a token sequence back into readable text. It is not always a matter of concatenation — whitespace and punctuation conventions have to be restored, and whether the original string can be recovered exactly depends on how the tokenizer was designed. Schemes that treat spaces as ordinary symbols exist precisely so that detokenization is deterministic rather than rule-driven guesswork.