#reward-hacking

3 articles

01 ·Paper Deep-Dives·★ MEMBER·PAPER·10 min read Paper Explained: False Frontiers — Diagnosing and Breaking Co-Cheating in Self-Evolving Search Agents When the model that writes questions and the model that answers them start agreeing on the same wrong answers, internal reward climbs while real accuracy doesn't. This paper measures that 'co-cheating' with an external audit and breaks it with CrossFit: let a solver that never trained on a source grade the questions from it. Mechanism, results, cost, and limits. 02 ·Agents·★ MEMBER·PAPER·9 min read CogEvol: What the Reward Cannot Measure, RL Will Quietly Destroy A technical report on a model family that generates teaching material in a single pass. Its centerpiece is an incident the authors disclose in full: a screenshot-only reward taught the policy to ship games that looked convincing and could not be played. 03 ·Evaluation & Judging·★ MEMBER·PAPER·10 min read Reward Hacking — Whatever You Measure Is Where It Breaks The moment you pick a metric, that metric starts to rot. This piece explains why Goodhart's law is statistically unavoidable, walks through real failures from boat races that spin in circles to RLHF verbosity, sycophancy and hardcoded unit tests, and covers how to detect the gap between optimization pressure and true performance.