Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code?

A walkthrough of SWE-bench Science, a repository-level benchmark of 119 tasks from 98 repos across 20 scientific domains. The best agent scores under 50% pass@1, and the paper names four recurring failure mechanisms plus an ablation showing scientific knowledge is not uniformly helpful.

SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

When the telescope lens has a hairline crack

Few people second-guess the numbers coming out of an observatory. But suppose the lens that produced them has a hairline crack. The record itself is quietly distorted — and the distortion does not announce itself as an error. It shows up as a plausible number.

In modern science, software plays the role of that lens. Seismic analysis, protein structure estimation, climate model integration — in each case the second half of "measuring with an instrument" is code. That is exactly where this paper starts: software increasingly functions as part of the scientific instrument itself, so failures in scientific code can compromise not only program behavior but also the evidence underlying scientific conclusions (Abstract).

A bug in a web app blanks the screen and somebody notices. A bug in scientific code nudges the slope of a figure and sails through review. That "failing quietly" property is what makes repairing scientific software a distinct problem.

What existing agent evaluations left out

Evaluation of AI coding agents has matured quickly. Hand an agent an entire repository, tell it to write a patch that fixes an issue, and score it on whether hidden tests pass — the SWE-bench family of setups (the breadth of that lineage is covered in the SWE-bench Pro Max walkthrough).

But, the paper argues, existing evaluations largely emphasize aggregate task success and give limited insight into why agents fail when repairing scientific software (Abstract).

That is a substantive point about evaluation design. A single line saying "45% passed" tells you nothing about whether the other 55% misread the problem or read it correctly and failed to finish the repair. Those two cases call for completely different next moves.

What SWE-bench Science is

So the authors built SWE-bench Science. Per the abstract, it consists of:

• 119 tasks
• drawn from 98 GitHub repositories
• spanning 20 scientific domains
• as a repository-level benchmark

Look at the ratio. 119 tasks from 98 repositories is roughly 1.2 tasks per repository. Rather than mining many problems from a few projects, the set is spread wide and thin. That design makes it hard for an agent to score by learning one project's conventions and riding it through a streak.

"Repository-level" is a heavier condition than it first sounds. Instead of writing a single function, the agent receives a codebase of tens of thousands of lines and must locate the offending function itself, fix it, and pass the existing tests without breaking them. The work is weighted toward investigating rather than writing — the same search loop covered in LLM agent fundamentals.

Three paradigms

Each task is organized into one of three paradigms (Abstract).

1. Issue-driven An existing issue is the starting point. "It crashes on this input," "this result disagrees with theory" — you work back from the report to the cause. This is closest to classic SWE-bench.

2. Expert-exploratory No explicit defect report to lean on; the agent uses domain expertise to go looking. This models situations where nobody has yet put the breakage into words.

3. Engineering-integration Not a single repair but making the pieces mesh as a system. Fixing the component is not enough; it has to line up with everything around it.

The three-way split is itself a claim: work on scientific software is not the single track of "bug report → fix." Tasks that require exploration and tasks that require integration are distinct kinds of hard.

The pass@1 yardstick

Before the results, a word on the metric. What follows is the standard definition of the metric, not a claim from the paper.

Symbol by symbol: $N$ is the number of tasks (119 here). $\mathbb{1}[\cdot]$ is a box that yields 1 if the statement inside holds and 0 otherwise. Summing and dividing by $N$ means the formula says one thing: the fraction of tasks fixed on the first try.

The "first try" matters. A metric that lets you sample repeatedly and count a single hit will report a friendlier number, but what a human actually reviews in production is the first patch that comes out. Treat pass@1 as the strict yardstick that matches operational reality.