Students Professional
JA EN

#swe-bench

8 articles

01 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: HarnessDev — Can an LLM Build and Maintain the System It Runs Inside? A ground-up walkthrough of HarnessDev, a benchmark that asks whether an LLM can build an agent harness from a deliberately useless seed and then improve it from execution feedback. Models can build one; the gains rarely survive contact with held-out tasks or a different runtime model. 02 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: EarlyEval — Making Agent Evaluation Cheaper by Stopping Early One evaluation pass over an agentic benchmark costs hundreds of dollars. EarlyEval exploits the fact that an agent's ending is legible from its middle, halting runs to cut 13–26% of steps. Mechanism, numbers, and limits, straight from the paper. 03 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: PILOT in the Loop — Fixing the Run While It Is Still Running Agent self-improvement that only starts after execution ends arrives too late. A ground-up walkthrough of PILOT, a supervisor–worker harness that redirects the live run while distilling skills from it, from intuition to equations to measured numbers and stated limits. 04 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points. 05 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 06 ·Agents·★ MEMBER·PAPER·9 min read Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code? A walkthrough of SWE-bench Science, a repository-level benchmark of 119 tasks from 98 repos across 20 scientific domains. The best agent scores under 50% pass@1, and the paper names four recurring failure mechanisms plus an ablation showing scientific knowledge is not uniformly helpful. 07 ·Agents·★ MEMBER·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 08 ·Agents·★ MEMBER·PAPER·7 min read Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.