refinedweb

Definition

A large pre-training dataset released by TII, built purely from web crawl data. Its headline claim is that aggressive filtering and deduplication alone can produce web-only data competitive with corpora that mix in curated sources such as books and papers — a direct challenge to the assumption that curation is what makes a corpus good. It is the dataset associated with the Falcon model family.