2026-08-12·FREE·PAPER·7 min read

Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring

A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.

#agents #benchmark #swe-bench #refactoring #coding-agent

ModalitytextTaskagents

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

Primary source — what this article is built on

undefined2026-08-10→undefined2026-08-12same month

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code RefactoringYuling Shi, Jinghan Xu, Kelin Fu et al. · 2026-08-10 · v1arXiv:2608.09802Paper page·PDF

undefined

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

When a benchmark is 75% solved, is it still measuring anything?

The standard yardstick for AI coding agents has been SWE-bench: hand the agent a real GitHub issue, let it modify the repository, and count the task as solved if the tests pass. According to the paper, frontier agents now exceed 75% on the curated SWE-bench Verified split, and the gaps between top systems have narrowed to the point where the benchmark struggles to tell them apart (§1).

The deeper problem is the grading itself. An audit cited by the paper found defects in roughly 60% of SWE-bench Verified's unsolved instances: 35.5% had overly narrow tests that reject genuinely correct solutions, and 18.8% had overly broad tests that check requirements never stated in the task. There is also evidence that frontier models can reproduce gold patches verbatim from training data — memorization, not capability — which led OpenAI to deprecate the benchmark entirely (§1).

SWE-Bench ProMax (published at COLM 2026) is a head-on answer. Its chosen arena is refactoring: restructuring code internally while leaving its externally visible behavior untouched.

An analogy: replacing a faucet vs. re-plumbing the whole house

Existing benchmarks mostly test bug fixes: 86% of SWE-bench Verified instances are resolved by editing a single file (§1). That is the "replace one leaky faucet" job.

Refactoring is "replace the plumbing throughout the house with a new standard — and afterwards, every tap must still deliver hot water exactly as before." You touch far more locations, and one stretch of old pipe left behind fails the whole job. In the paper's flagship example, reorganizing the headers of NASA's F´ (fprime) flight-software framework took coordinated changes across 244 files (§1, Appendix C). What gets tested is coordinated multi-file editing plus behavior preservation — among the heaviest skills in professional software work.

How scores drift away from ability

Benchmark saturation and contamination follow the shape of a familiar machine-learning failure: overfitting. Performance on problems the model has effectively seen keeps rising while performance on genuinely novel ones does not. The more a public benchmark leaks into training data, the more its score measures "have I seen this?" rather than "can I do this?".

FIG 1Raise the polynomial degree and training error keeps falling while test error pulls away — an analogy for how public-benchmark scores drift away from real capability

ProMax attacks this drift with three design decisions: (1) issue descriptions are rewritten from scratch to remove ambiguity and answer leakage, (2) test suites are manually reviewed to remove overly narrow and overly broad tests, and (3) tasks are mined from fresh commits made after January 2025 (§3.2).

From 29,782 candidates down to 170 survivors

Each instance is a four-piece kit (§3.1): (1) a Docker environment holding the repository at the commit just before the refactoring, dependencies installed; (2) an issue description specifying the intended change in precise natural language; (3) a test suite for verification; and (4) the original developer's gold patch. The agent receives only the first two and works autonomously.

Construction runs in three stages (§3.2). Post-January-2025 commits containing "refactor" are collected via the GitHub API from open-source repositories with 500+ stars; a Docker environment is built for each candidate and the gold patch mechanically verified to pass the tests; then human experts, assisted by LLMs, filter out tasks that are single-file or too simple, strip overly narrow and overly broad tests, and rewrite every issue description from scratch — each verified to be a necessary and sufficient condition for the gold patch. Of 29,782 initial candidates, 170 instances survive: roughly 1 in 175.

The survivors operate at a different scale from prior benchmarks (§3.3). Gold patches average 11.4 source files and 261.6 lines of code (up to 182 files), or 15.9 files including test changes. Thirty percent of instances modify more than 10 files and 32% change over 200 lines (§1). The tasks span Python, Java, TypeScript, Go, C, C++, and Rust, drawn from 70 repositories including Django, Kubernetes, Angular, and OpenSSL (§3.3, Appendix A).

Grading is strictly outcome-driven: only the final state of the repository is evaluated, never the steps taken (§3.1).

\mathrm{resolved}(i) = \mathbb{1}\Big[\,\forall\, t \in T_i:\ \mathrm{pass}\big(t,\ \mathrm{repo}_i^{\mathrm{final}}\big)\Big]

(1)

Stated in words: instance $i$ counts as resolved (value 1) only if every test $t$ in its suite $T_i$ passes on the agent's final repository; one failing test makes it 0. The symbol $\mathbb{1}[\cdot]$ is the indicator function — 1 when the condition inside is true, 0 otherwise.

\mathrm{Pass@1} = \frac{1}{N}\sum_{i=1}^{N} \mathrm{resolved}(i), \qquad N = 170

(2)

The fraction of instances resolved in a single attempt — the paper's primary metric (§4.1).

Results: the best model manages 41.2%

Six models are evaluated — Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2, GLM-5, Kimi-K2.5, and Qwen3.5 (§4.2) — each under two scaffolds, the minimal mini-swe-agent and the tool-rich OpenHands, capped at 300 steps and $10 per instance (§4.1). Four findings stand out (§5.1).

1. Everyone struggles. The best result is GPT-5.2 under OpenHands at 41.2%. Against the 75%+ posted on SWE-bench Verified, the gap shows how unsolved multi-file, behavior-preserving change still is.

2. Open-weight models get close at a twentieth of the cost. Claude Sonnet 4.6 spends $4.77 per instance for 38.8%; GLM-5 reaches 36.5% at $0.24. Paying more does not buy proportionally more.

3. The scaffold changes everything. GPT-5.2 scores 21.8% under mini-swe-agent and 41.2% under OpenHands — nearly double with the same model. Every model except Gemini-3-Pro improves on the richer scaffold.

4. Language strengths are scattered. Under OpenHands, Claude Sonnet 4.6 leads on TypeScript (53.6%) and Rust (63.6%), GPT-5.2 tops Python (48.3%) and C (75.0%), while Gemini-3-Pro scores 0.0% on TypeScript. The paper reads this variance as likely reflecting training-data composition rather than inherent language difficulty (§5.1).

Anatomy of failure: agents don't finish the job

The most instructive part of the paper is how agents fail (§5.2). Comparing files modified against the gold patch, the distributions match up to about 5 files, but where the gold-patch distribution reaches its 90th percentile only around 20 files, the agents analyzed (Claude Sonnet 4.6 and Kimi-K2.5) plateau at roughly 10. Crucially, this is not a failure to find the right files: agents locate and correctly edit the core files, then stop short of propagating the change to peripheral call sites, configuration, test fixtures, and documentation. The leftover downstream inconsistencies fail the tests. The paper names this incomplete refactoring and identifies it as the dominant failure mode.

The second symptom is behavioral. Successful runs converge in few interaction rounds; failed runs cycle through read–edit–fail–revert, burning steps without widening the scope of their changes — what the paper calls unproductive exploration (§5.3). Qwen3.5, which takes the most steps of any model (155.4 on average under mini-swe-agent), finishes last on that scaffold at 20.6%: persistence alone is not progress.

FIG 2How different growth orders scale with input size n. Even if the number of files grows linearly, the web of consistency constraints between them grows much faster — an intuition for why multi-file coordination changes the nature of the difficulty

The bottleneck, the paper concludes, is not raw reasoning on any single edit but the capacity to hold a coherent plan across many files and follow the cascading consequences of a structural change all the way to the end (§5.2).

Limitations, and how to read the numbers

The 28 TypeScript instances come from just 2 repositories — 25 from Angular — so that column is closer to an Angular score (Appendix A). Per-language samples are 20–29 instances, so one instance swings a score by 3–5 points. Curation was human-led but LLM-assisted, and the task-category labels were produced by Claude Sonnet 4.6 (analysis only; grading is unaffected) (Appendix B). Evaluation is single-shot Pass@1; gains from retries are unmeasured (§4.1). And — my own note, not the paper's — mining post-2025 public commits lowers contamination risk today but cannot guarantee resistance for future models trained on the same public repositories. Per Appendix B, 41.2% of instances also involve bug fixes and 43.5% new features, so this measures compound engineering work, not refactoring in isolation.

How this is used in the field

When delegating a large refactor to an agent — a tech lead handing a rename or API migration to a Claude Code / Cursor / OpenHands-style agent — knowing that the dominant failure is stopping halfway changes how you operate. The direct countermeasure: before starting, mechanically enumerate every affected site with grep or LSP reference search and embed the list in the task description as a checklist. Test fixtures, config files, and docs are the sites most often left behind — that is your inspection list.

When choosing models and tooling, three things follow from the measurements: scaffold richness can matter more than model choice; the expensive model is not always the better buy ($0.24 GLM-5 lands within 2.3 points of $4.77 Claude Sonnet 4.6); and different models lead on different languages. In internal evaluations, always set step and cost ceilings (the paper used 300 steps and $10) — failed runs consume the most steps, so an uncapped setup bills you hardest for failure.

For evaluators and researchers, the habit worth stealing from the methodology (§3.2) is the checklist itself: are the tests reviewed for being too narrow or too broad? Is the task description a necessary and sufficient condition for the reference solution? How many files does the task really touch?