Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring
A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Primary source — what this article is built on
undefined2026-08-10→undefined2026-08-12same month
SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code RefactoringYuling Shi, Jinghan Xu, Kelin Fu et al. · 2026-08-10 · v1arXiv:2608.09802Paper page·PDFundefined
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.
When a benchmark is 75% solved, is it still measuring anything?
The standard yardstick for AI coding agents has been SWE-bench: hand the agent a real GitHub issue, let it modify the repository, and count the task as solved if the tests pass. According to the paper, frontier agents now exceed 75% on the curated SWE-bench Verified split, and the gaps between top systems have narrowed to the point where the benchmark struggles to tell them apart (§1).
The deeper problem is the grading itself. An audit cited by the paper found defects in roughly 60% of SWE-bench Verified's unsolved instances: 35.5% had overly narrow tests that reject genuinely correct solutions, and 18.8% had overly broad tests that check requirements never stated in the task. There is also evidence that frontier models can reproduce gold patches verbatim from training data — memorization, not capability — which led OpenAI to deprecate the benchmark entirely (§1).
SWE-Bench ProMax (published at COLM 2026) is a head-on answer. Its chosen arena is refactoring: restructuring code internally while leaving its externally visible behavior untouched.
An analogy: replacing a faucet vs. re-plumbing the whole house
Existing benchmarks mostly test bug fixes: 86% of SWE-bench Verified instances are resolved by editing a single file (§1). That is the "replace one leaky faucet" job.
Refactoring is "replace the plumbing throughout the house with a new standard — and afterwards, every tap must still deliver hot water exactly as before." You touch far more locations, and one stretch of old pipe left behind fails the whole job. In the paper's flagship example, reorganizing the headers of NASA's F´ (fprime) flight-software framework took coordinated changes across 244 files (§1, Appendix C). What gets tested is coordinated multi-file editing plus behavior preservation — among the heaviest skills in professional software work.
How scores drift away from ability
Benchmark saturation and contamination follow the shape of a familiar machine-learning failure: overfitting. Performance on problems the model has effectively seen keeps rising while performance on genuinely novel ones does not. The more a public benchmark leaks into training data, the more its score measures "have I seen this?" rather than "can I do this?".
ProMax attacks this drift with three design decisions: (1) issue descriptions are rewritten from scratch to remove ambiguity and answer leakage, (2) test suites are manually reviewed to remove overly narrow and overly broad tests, and (3) tasks are mined from fresh commits made after January 2025 (§3.2).
From 29,782 candidates down to 170 survivors
Each instance is a four-piece kit (§3.1): (1) a Docker environment holding the repository at the commit just before the refactoring, dependencies installed; (2) an issue description specifying the intended change in precise natural language; (3) a test suite for verification; and (4) the original developer's gold patch. The agent receives only the first two and works autonomously.
Construction runs in three stages (§3.2). Post-January-2025 commits containing "refactor" are collected via the GitHub API from open-source repositories with 500+ stars; a Docker environment is built for each candidate and the gold patch mechanically verified to pass the tests; then human experts, assisted by LLMs, filter out tasks that are single-file or too simple, strip overly narrow and overly broad tests, and rewrite every issue description from scratch — each verified to be a necessary and sufficient condition for the gold patch. Of 29,782 initial candidates, 170 instances survive: roughly 1 in 175.
Comments
Sign in to comment