Students Professional
JA EN

#reinforcement-learning

26 articles

01 ·Agents·★ MEMBER·PAPER·9 min read CogEvol: What the Reward Cannot Measure, RL Will Quietly Destroy A technical report on a model family that generates teaching material in a single pass. Its centerpiece is an incident the authors disclose in full: a screenshot-only reward taught the policy to ship games that looked convincing and could not be played. 02 ·Training & Alignment·★ MEMBER·PAPER·13 min read Paper Walkthrough: It Takes Two to Match — Co-Evolving Both Sides of Retrieval with RL A paper that rebuilds the first stage of search by having two LLMs write keywords — one for queries, one for items — and training them in alternation until their vocabularies meet. Explained from scratch. 03 ·Agents·★ MEMBER·PAPER·11 min read Paper Explained: What Makes Good Agentic Data? The ACE Lens A survey that recasts agentic training data as a four-part object (environment, task, interaction, verifier) and reframes generation as constrained distribution design: admit on Accuracy, place mass by Complexity, spread coverage with divErsity. 04 ·Agents·★ MEMBER·PAPER·13 min read Paper Walkthrough: UI-Venus-2 — Taking Screen-Operating Agents From Benchmarks to Real Work A ground-up read of the UI-Venus-2 Technical Report: a GUI agent that drives phones, browsers and desktops from pixels alone. What it scales, how it verifies its own training data, and where it still falls short. 05 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: Training Agents to Evolve with Their Harness Why compact models break when you rewrite prompts and tool definitions every week, and what the team behind Taobao Live's AI avatar streamers does about it. A ground-up walkthrough of Harness-Aware Training, from intuition to equations to measured numbers and stated limits. 06 ·★ MEMBER·PAPER·12 min read Paper walkthrough: Qwen-Drive-1.0 — bolting 3D perception and planning onto a VLM without touching its architecture Leave the pretrained VLM exactly as it is, attach a BEV perception head and a Planning Expert, and you get 3D detection, occupancy, maps and trajectory generation in one model — general vision-language ability nearly intact, 90.7 PDMS on NAVSIM. A ground-up walkthrough, straight from the paper. 07 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement A third to a half of the teacher's token-level grades are wrong, yet the student improves just as fast either way. This paper traces on-policy distillation's gains not to imitating a teacher but to suppressing the student's own low-probability tokens — and drops the teacher entirely. 08 ·Training & Alignment·★ MEMBER·PAPER·10 min read Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward. 09 ·★ MEMBER·PAPER·20 min read Lucida, explained — turning a room video back into parts you can move, by deferring precision to the end A system that rebuilds a real indoor scene from video as individually movable 3D assets. It keeps the parse–generate–place order but redistributes what each step is allowed to demand, and hands the final placement to a VLM policy that operates a 3D editor in a closed loop. 10 ·Inference & Serving·★ MEMBER·PAPER·11 min read TTPO Explained: Training a Model Mid-Exam, With No Answer Key Majority-vote pseudo-labels are wrong on roughly 85% of competition math prompts. So why does training still work? A ground-up walkthrough of TTPO's asymmetric design: distill the rollouts that agree, penalize the ones that disagree. 11 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Walkthrough — WarpSAC: When RL's Safety Rails Become Handcuffs Once GPU-parallel simulators flood the replay buffer, SAC's normalization and clipped double-Q flip from helpful to restrictive. The paper isolates three design axes and prescribes removing stabilizers — not stacking them — when data are abundant. 12 ·Distillation & Compression·★ MEMBER·PAPER·9 min read On-Policy Distillation — Learning From What the Student Actually Writes Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD. 13 ·Paper Deep-Dives·★ MEMBER·PAPER·13 min read Paper Walkthrough: Turning Game Development into a Verifiable Trajectory Data Engine — RLHEV and AWoMo A paper arguing that what world models lack is not data or compute but cheap grading. We work through RLHEV — post-training on game-engine checks fused with developer accept/reject decisions — from first principles to equations, interactive figures, and the reported numbers. 14 ·Paper Deep-Dives·★ MEMBER·PAPER·11 min read Paper walkthrough: Apodex 1.1 — scaling agents around completed work Not a bigger model and not more thinking time — Apodex 1.1 scales two other surfaces: the environments an agent learns in, and the way work is organised across agents. A walkthrough from the task contract to the AgentOS delivery gate, the numbers, and the limits. 15 ·★ MEMBER·PAPER·9 min read Paper Walkthrough: Annotations as Rollouts — Dropping the Ground Truth Into the Group as a Ninth Answer RL post-training for video MLLMs starves because sampled rollouts almost never contain the right answer. OraRL adds the annotation itself to the group as an extra rollout, then repairs the advantage inversion that naive mixing causes. 16 ·Agents·★ MEMBER·PAPER·9 min read AlphaGo from Scratch — The Marriage of Search and Learning Starting from why Go was considered unsolvable for so long, this piece unpacks how the policy network, the value network, Monte Carlo tree search and self-play each cover the others' weaknesses — with the formulas and the code. It closes with what this design handed down to inference-time compute in LLMs. 17 ·Agents·★ MEMBER·PAPER·9 min read Paper Explained: EnvHarness — Reshaping an Agent's Training World Without Rebuilding It The environments LLM agents train in are hand-built and frozen: blind to the agent's weaknesses and left behind as it improves. EnvHarness wraps a static environment in plug-in components that reshape its behavior without touching the underlying logic — and keep the original verifier intact. 18 ·Agents·★ MEMBER·PAPER·12 min read Paper Explained: Co-RL — Reasoning Without Labels, Emerging From a Diverse Cohort Grade your own answers long enough and the model collapses. Co-RL breaks that loop by rewarding each agent against a peer's majority vote, matching supervised training without touching a single ground-truth label. The mechanism, the dynamics, the numbers, and the traps — straight from the paper. 19 ·Inference & Serving·★ MEMBER·PAPER·15 min read Paper Explained: Agentic ESOpt — Drop Backprop, Jiggle the Weights, and Train Long-Horizon LLM Agents No gradients at all: spawn G perturbed copies of the model, run them in the environment, and nudge the weights toward the ones that scored well. That buys full-parameter updates of a 27B agent at inference-level memory (8.41GB) — and a 12.50-point lead over GRPO on 15-move Sudoku. A ground-up walkthrough of the NUS-led paper. 20 ·Agents·★ MEMBER·PAPER·13 min read Paper Deep-Dive: Recursive Synthesis — Extending Verified Tasks Into 40,000 Long-Horizon Terminal Problems Long-horizon terminal tasks cost hundreds to thousands of dollars each to author. RST mass-produces them for about $0.05 by recursively extending already-verified seeds. Fifteen rounds yielded 37,484 tasks, with oracle solutions growing from a median of 67 lines to 374, and the resulting trajectories genuinely improved Qwen3.5 under SFT and PPO — a walkthrough of the data-synthesis paper from Tencent and collaborators. 21 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough: Qwen-UI-Agent — How Alibaba Built a GUI Agent That Works on Real Phones and PCs A guided tour of Alibaba MAI-UI team's technical report on Qwen-UI-Agent: a foundation GUI agent trained on 100+ physical phones, a hybrid GUI+CLI action space, and online RL over 100+ turn trajectories, reaching 92.2% on a real-device benchmark. 22 ·Agents·★ MEMBER·PAPER·11 min read Paper Deep-Dive: Frontis-MA1 — Training the AI That Builds AI: One Step Toward Recursive Self-Improvement in ML Engineering Instead of training a model to solve tasks, train the operations that improve solutions — then run evolutionary search with those trained operations. A walkthrough of the OpenMLE stack and the 35B Frontis-MA1 model, straight from the paper: what's behind the 39.39%→71.21% jump on MLE-Bench Lite, how the search got cheaper and better at once, and the limits the authors themselves spell out. 23 ·★ MEMBER·PAPER·9 min read Paper explained: DEFT-RLVR — show a driving VLM the future trajectory too early and it fabricates its reasoning Let a teacher model peek at the ground-truth trajectory while writing chain-of-thought annotations for driving, and severe hallucinations double from 29% to 50% — the paper calls this trajectory anchoring bias. This article walks through AD-MCQ, which turns planning into a verifiable multiple-choice task, and DEFT-RLVR (arXiv:2608.01755), which reveals trajectories only after the model commits to a decision, using nothing but the paper itself as the source. 24 ·Training & Alignment·★ MEMBER·PAPER·11 min read Paper Deep-Dive: ABSeeker — Training Long-Horizon Search Agents by Grading Each Step Backward from the Answer When an agent chains dozens of web searches, the only reward is whether the final answer was right — so every good move and every bad move along the way gets graded identically. We walk through ABC (Answer-Backtracked Credit Assignment), which recovers clues by working backward from the answer and scores every step against them, and ABSeeker, the 4B model trained with it — all from the paper itself. 25 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: EnvACE — Agents That Rehearse the World Instead of Calling It EnvACE trains a tool-using LLM agent without ever calling an external environment: the same policy alternates between acting and playing the environment. We walk through role-wise GRPO, test-time private rehearsal, results on four agent benchmarks, and the limitations. 26 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Deep-Dive: AgentOPSD — Finding the Turn That Won the Game with Recursive Bayesian Belief Updates When a long multi-turn episode only tells you 'success' or 'failure' at the very end, which turn actually mattered? AgentOPSD reinterprets self-distillation gaps as Bayesian evidence and recursively updates a success belief in log-odds space. We walk through the mechanism, the 89.1% ALFWorld result, the ablations, and the limitations — all from the paper itself.