deduplicating

Definition

Removing duplicate and near-duplicate documents from a training corpus. Web-scale scrapes contain the same text many times over, and leaving it in wastes compute, encourages verbatim memorization, and risks contaminating benchmarks whose items also appear online. Practical pipelines go beyond exact matching, using hashing schemes such as MinHash to catch near-duplicates that differ only in boilerplate or formatting.