resolve

Definition

To close out a task so that the stated requirement is actually satisfied. In agentic software benchmarks it has a precise meaning: an issue counts as resolved only if the model's patch makes the hidden tests pass, and the headline metric is the resolve rate. The judgement is binary — partial credit for plausible-looking code is exactly what this metric refuses to give.

Explainers using this term

• A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually MeasureSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code?SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
• Evaluating Agents — How Benchmarks and Harnesses Are BuiltSWE-bench: Can Language Models Resolve Real-World GitHub Issues?