#pretraining
7 articles
01
·Large Language Models·★ MEMBER·PAPER·13 min read
Paper walkthrough: Puro-2B — pretraining a 2B model from scratch for $6.9K on consumer GPUs
A team ran 1.4 trillion tokens of pretraining on gaming GPUs and reached Qwen2-1.5B-level quality for roughly $4.4K. Here is the cost structure, the FP8 accounting, the effective learning rate, and the curriculum averaging — from first principles.
02
·Machine Learning Basics·★ MEMBER·PAPER·11 min read
Self-Supervised Learning — The Day Unlabeled Data Became an Asset
Nobody has to label the data — the data can write its own exam. A ground-up tour of the two families (masked prediction and contrastive learning), from the intuition through the InfoNCE loss, an interactive figure, PyTorch code, and finally why LLM pretraining is the largest self-supervised system ever run.
03
·Machine Learning Basics·★ MEMBER·PAPER·11 min read
Self-Supervised Learning — The Day Unlabeled Data Became an Asset
Nobody has to label the data — the data can write its own exam. A ground-up tour of the two families (masked prediction and contrastive learning), from the intuition through the InfoNCE loss, an interactive figure, PyTorch code, and finally why LLM pretraining is the largest self-supervised system ever run.
04
·Large Language Models·★ MEMBER·PAPER·10 min read
Building a Pretraining Corpus — From Web Sludge to Textbook Quality
Behind the single line "pretrained on a large corpus of web text" sit four stages: text extraction, quality filtering, deduplication, and mixture weights. This is a from-scratch walkthrough of how a gravel heap called Common Crawl gets sifted into textbook-quality prose — from the MinHash equation to the parameter names you actually touch.
05
·Large Language Models·★ MEMBER·PAPER·10 min read
Building a Pretraining Corpus — From Web Sludge to Textbook Quality
Behind the single line "pretrained on a large corpus of web text" sit four stages: text extraction, quality filtering, deduplication, and mixture weights. This is a from-scratch walkthrough of how a gravel heap called Common Crawl gets sifted into textbook-quality prose — from the MinHash equation to the parameter names you actually touch.
06
·Training & Alignment·★ MEMBER·PAPER·11 min read
Building a Dataset in Practice — Collect, Clean, Blend
Most of the work in building a model is really the work of building its dataset. From designing the population you sample from, to calibrating filter thresholds, measuring leakage into your test set, converting mixture ratios into epochs, using synthetic data where it belongs, and writing annotation guidelines that actually hold — explained from zero, with the formulas and the procedures.
07
·★ MEMBER·PAPER·10 min read
Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?
What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget.