#pipeline-parallel

2 articles

01 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Walkthrough: Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training — training the draft alongside a 122B policy at 256K tokens Rollout generation dominates RL post-training time. NVIDIA speeds it up with speculative decoding and keeps training the draft model alongside the policy — and shows how to get past the two walls that context parallelism and pipeline parallelism put in the way. A from-scratch walkthrough grounded in the paper text. 02 ·Parallel & Distributed·★ MEMBER·10 min read Distributed Training from Scratch — Data Parallel, Model Parallel, and When Communication Becomes the Bottleneck Why one machine is not enough, counted out in bytes; data parallelism and all-reduce; what ZeRO and FSDP actually shard; tensor and pipeline parallelism. Then the ratio of computation to communication that tells you where scaling stops paying — and gradient accumulation, NCCL settings and how to diagnose a hang.