#speculative-decoding

2 articles

01 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Walkthrough: Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training — training the draft alongside a 122B policy at 256K tokens Rollout generation dominates RL post-training time. NVIDIA speeds it up with speculative decoding and keeps training the draft model alongside the policy — and shows how to get past the two walls that context parallelism and pipeline parallelism put in the way. A from-scratch walkthrough grounded in the paper text. 02 ·Inference & Serving·FREE·PAPER·8 min read Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single Output Speculative decoding lets a small model draft several tokens ahead while a large model verifies them in one batch — provably identical outputs, 2–3x faster in practice. A from-scratch walkthrough of the mechanism and the intuition behind the acceptance rate α, straight from the original paper.