#scaling
3 articles
01
·Model Families·★ MEMBER·9 min read
The GPT Lineage — Design Thinking from GPT-1 to Today
A generation-by-generation walk from GPT-1 to GPT-4o and the reasoning models, asking what changed and what deliberately did not. At the centre sits one machine that only ever predicts the next token; what moved was scale, the order of training, and how the output was disciplined.
02
·★ MEMBER·PAPER·10 min read
Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?
What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget.
03
·Paper Deep-Dives·★ MEMBER·PAPER·9 min read
Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer
A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.