pile

Definition

In everyday English a heap, but in this field it nearly always names The Pile, the open English text corpus released by EleutherAI for language-model pre-training. Its distinguishing feature is composition: rather than scraped web text alone, it bundles many labelled sub-corpora — academic papers, code, books, dialogue, web — so researchers can reason about what a model was fed. It became a common baseline dataset for open-weight models.

Explainers using this term

• Building a Pretraining Corpus — From Web Sludge to Textbook QualityThe Pile: An 800GB Dataset of Diverse Text for Language Modeling