swe-bench

Definition

A benchmark built from real issues in real GitHub repositories: the model must produce a patch, and it is scored by whether the repository's own tests pass afterwards. Because grading is execution-based rather than similarity- or preference-based, it is a much harder and more honest measure than earlier code benchmarks. It has become the headline number for coding agents, since solving an issue requires navigating an unfamiliar codebase, not just writing one function.

Explainers using this term

• A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually MeasureSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code?SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
• Evaluating Agents — How Benchmarks and Harnesses Are BuiltSWE-bench: Can Language Models Resolve Real-World GitHub Issues?