swe-bench

Definition

A benchmark built from real issues in real GitHub repositories: the model must produce a patch, and it is scored by whether the repository's own tests pass afterwards. Because grading is execution-based rather than similarity- or preference-based, it is a much harder and more honest measure than earlier code benchmarks. It has become the headline number for coding agents, since solving an issue requires navigating an unfamiliar codebase, not just writing one function.

Explainers using this term

• A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually MeasureSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Evaluating Agents — How Benchmarks and Harnesses Are BuiltSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual RefactoringSWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring