Students Professional
JA EN

#grpo

16 articles

01 ·Training & Alignment·★ MEMBER·PAPER·13 min read Paper Walkthrough: It Takes Two to Match — Co-Evolving Both Sides of Retrieval with RL A paper that rebuilds the first stage of search by having two LLMs write keywords — one for queries, one for items — and training them in alternation until their vocabularies meet. Explained from scratch. 02 ·Inference & Serving·★ MEMBER·PAPER·14 min read Paper Walkthrough: From Production Traffic to Post-Training — Folding 200 Internal Apps Into One Self-Hosted LLM How a company forced to self-host by data-residency rules collapsed a sprawling model fleet into one checkpoint: classify production failures by hand, train one GRPO expert per weak axis, then merge the weights with SLERP. Includes three real reward-hacking failures. 03 ·Training & Alignment·★ MEMBER·PAPER·10 min read Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward. 04 ·Paper Deep-Dives·★ MEMBER·PAPER·12 min read Paper Walkthrough — J-Zero: Growing the Challenger, the Solver, and the Judge Together from Zero Data A model that writes its own problems, solves them, and grades them — with no external data and no human labels. J-Zero's twist is that the grader learns too, which is what lets it keep improving past the two-iteration wall where prior methods stall. 05 ·★ MEMBER·PAPER·11 min read Paper Walkthrough: TLive-Omni — An Omni-Modal Model That Watches, Listens, and Tells You the Timestamp A from-scratch walkthrough of TLive-Omni, an omni-modal understanding model built for e-commerce live streaming: Per-vGrid timestamped audio-video interleaving, a three-stage SFT recipe, and Faithful-RFT, which rewards answer faithfulness instead of visible reasoning — grounded strictly in the paper's own numbers. 06 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: Embodied-Navigator (TAMP-Nav) — Let the VLM Just Point, and Navigation Gets Both Faster and Better Instead of asking a VLM for 3D coordinates, have it point at a 2D pixel; think and store memory only at key nodes; align it all with two-level GRPO. A ground-up walkthrough of the design that hits 66.2% SR on R2R-CE at 16.58s per task using only 90k training trajectories. 07 ·Training & Alignment·★ MEMBER·PAPER·13 min read DPO and What Came After — The Lineage That Simplified RLHF Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have. 08 ·★ MEMBER·PAPER·9 min read Paper Walkthrough: Annotations as Rollouts — Dropping the Ground Truth Into the Group as a Ninth Answer RL post-training for video MLLMs starves because sampled rollouts almost never contain the right answer. OraRL adds the annotation itself to the group as an extra rollout, then repairs the advantage inversion that naive mixing causes. 09 ·Agents·★ MEMBER·PAPER·12 min read Paper Explained: Co-RL — Reasoning Without Labels, Emerging From a Diverse Cohort Grade your own answers long enough and the model collapses. Co-RL breaks that loop by rewarding each agent against a peer's majority vote, matching supervised training without touching a single ground-truth label. The mechanism, the dynamics, the numbers, and the traps — straight from the paper. 10 ·Large Language Models·★ MEMBER·PAPER·11 min read Paper Walkthrough: SA-MRPO — Stop Studying the Subject You've Already Aced When you train with several reward objectives at once, gradient budget keeps flowing to objectives that are already solved. SA-MRPO (arXiv:2608.16072) measures how saturated each objective is and discounts its weight accordingly, redirecting optimization toward whatever headroom remains. A walkthrough grounded strictly in the paper. 11 ·Agents·★ MEMBER·PAPER·9 min read Paper Explained: Video-DeepResearch — Agents That Watch a Video, Then Chase Down Every Lead A walkthrough of Video-DeepResearch, which pulls visual clues out of a video and corroborates them with web search. We follow the primary source to see how two failure modes — a modality bias that makes models dodge visual tools, and knowledge leakage that lets them answer from memory alone — are beaten with stage-wise tool unlocking and a two-phase SFT + GRPO recipe. 12 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: ToolArtist — Search, Draw, or Redraw? The Image Agent That Decides for Itself Drawing "a harvester collecting dragon's blood resin on Socotra in 1955" correctly takes less artistic skill than research. ToolArtist makes searching, reasoning, and drawing all actions of a single policy — a fully agentic image generator. We walk through the conversion trick behind its SFT data, the dual-reward RL scheme (RAD-GRPO), and the 0.79 WISE result, straight from the paper. 13 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training. 14 ·Training & Alignment·★ MEMBER·PAPER·11 min read Paper Deep-Dive: ABSeeker — Training Long-Horizon Search Agents by Grading Each Step Backward from the Answer When an agent chains dozens of web searches, the only reward is whether the final answer was right — so every good move and every bad move along the way gets graded identically. We walk through ABC (Answer-Backtracked Credit Assignment), which recovers clues by working backward from the answer and scores every step against them, and ABSeeker, the 4B model trained with it — all from the paper itself. 15 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: EnvACE — Agents That Rehearse the World Instead of Calling It EnvACE trains a tool-using LLM agent without ever calling an external environment: the same policy alternates between acting and playing the environment. We walk through role-wise GRPO, test-time private rehearsal, results on four agent benchmarks, and the limitations. 16 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Deep-Dive: AgentOPSD — Finding the Turn That Won the Game with Recursive Bayesian Belief Updates When a long multi-turn episode only tells you 'success' or 'failure' at the very end, which turn actually mattered? AgentOPSD reinterprets self-distillation gaps as Bayesian evidence and recursively updates a success belief in log-odds space. We walk through the mechanism, the 89.1% ALFWorld result, the ablations, and the limitations — all from the paper itself.