issues

Definition

Defects or open problems in an existing method, benchmark, or evaluation setup. A paper foregrounding issues is typically critical rather than constructive: it shows that something widely relied on — a leaderboard, a metric, a training corpus — is broken or contaminated in a way that undermines conclusions drawn from it. Such work is cited less than method papers but sets the conditions under which their claims can be believed.

Explainers using this term

• A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually MeasureSWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Evaluating Agents — How Benchmarks and Harnesses Are BuiltSWE-bench: Can Language Models Resolve Real-World GitHub Issues?