#swe-bench

8 articles

01 ·Large Language Models·★ MEMBER·PAPER·12 min read Paper walkthrough: RealSWE — re-measuring coding agents when the request is just "fix it" A paper that measures, on real data, how far SWE-bench problem statements sit from actual user requests, then builds 381 task families where only the information content and writing style change. Resolution rates fall 6.4 pp on average — and the ranking flips. 02 ·Agents·★ MEMBER·PAPER·12 min read HarnessDev explained (arXiv:2609.01437) — agent harness benchmark Identical weights, different harness: GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI. HarnessDev asks whether a model can build its own harness from a useless seed — and what survives held-out tasks. 03 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: EarlyEval — Making Agent Evaluation Cheaper by Stopping Early One evaluation pass over an agentic benchmark costs hundreds of dollars. EarlyEval exploits the fact that an agent's ending is legible from its middle, halting runs to cut 13–26% of steps. Mechanism, numbers, and limits, straight from the paper. 04 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: PILOT in the Loop — Fixing the Run While It Is Still Running Agent self-improvement that only starts after execution ends arrives too late. A ground-up walkthrough of PILOT, a supervisor–worker harness that redirects the live run while distilling skills from it, from intuition to equations to measured numbers and stated limits. 05 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points. 06 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 07 ·Agents·FREE·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 08 ·Agents·FREE·PAPER·7 min read Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.