Evaluating Agents — How Benchmarks and Harnesses Are Built

An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

What does "50% on SWE-bench" actually claim?

Suppose a launch deck says "SWE-bench Verified: 50%." Read plainly, that means the system can fix half of real GitHub bugs. But the number is not a property of the model. It is produced by four things together — the model, the scaffolding (the harness), the working environment, and the grading rule — and three of them never appear in the bar chart.

Think of hiring. Someone who aces a multiple-choice written test and someone who is handed a repository and told "please fix this bug" are being measured on different things. The second job rewards knowing how to run the test suite, not breaking a neighbouring feature while fixing yours, and backing out when you are stuck. Agent evaluation is trying to measure that second thing, which is exactly why the measurement itself becomes a design problem.

The three questions you ask of any benchmark — what was measured, how was it graded, and did the questions leak — all still apply. What follows is what gets added on top.

Three ways this differs from a quiz

1. There is an environment. The input to a multiple-choice question is a string. The input to an agent is a repository, a shell, a filesystem. It is not just larger — the agent can rewrite its own input. In the extreme, it can delete the failing test and report that everything passes. Whoever grades has to control both the initial and the final state of that environment.

2. There is a horizon. A single generation is one roll of the dice; an agent takes dozens of steps. Write $p$ for the per-step probability of not derailing and $n$ for the number of steps a task needs:

Here $p_i$ is the chance that step $i$ does not derail, $n$ is how many steps the task takes, and $P_{\text{success}}$ is the chance of finishing. Put in words: the end-to-end success rate is the per-step survival rates multiplied together — an obvious statement.

It stops being obvious once you put numbers in. At 99% per step over 50 steps, $0.99^{50}\approx0.61$; at 95%, $0.95^{50}\approx0.077$. Four points per step becomes almost an 8× gap end to end. Turned around: staring at the end-to-end number alone will never tell you which step is short.

3. There are side effects. Grading a quiz ends with checking an answer key. An agent writes to the world: it deletes files, runs commands, calls APIs. "The eval environment held production credentials" is an accident that happens before grading is even a question. An agent evaluation environment has to be built as a disposable box you are willing to destroy.

Opening up one SWE-bench instance

SWE-bench (Jimenez et al., 2023) is built mechanically from real GitHub issues and the commits that actually closed them. It contains 2,294 tasks drawn from 12 Python repositories, and one instance is made of these parts:

{
  "repo": "django/django",
  "base_commit": "…",        // state just before the fix; work starts here
  "problem_statement": "…",  // the issue text — in principle the only input the model gets
  "patch": "…",              // the fix a human actually wrote (reference; never shown)
  "test_patch": "…",         // the tests that landed alongside that fix
  "FAIL_TO_PASS": ["…"],     // tests that fail before and must pass after
  "PASS_TO_PASS": ["…"]      // tests that must keep passing either way
}

The design decision that matters is that correctness is not scored by similarity to the human patch. There are many valid ways to fix a bug, so string comparison would mark correct fixes wrong. What replaces it is execution: FAIL_TO_PASS carries "it is fixed" and PASS_TO_PASS carries "nothing else broke." That second half is what rejects a patch that works and takes a neighbouring feature down with it.

Learn the subset names too. Lite (300 instances) is a re-selected subset that is cheaper to run; Verified (500 instances) was checked instance by instance by people, dropping issues whose text under-specifies the fix and tests so strict that a correct fix still fails. Before comparing numbers, check which set produced them — with a different population, the same "50%" is a different quantity.

"resolved" is defined as a conjunction

SWE-bench's headline metric, the resolve rate, is 0 or 1 per instance:

$\mathrm{F2P}$ and $\mathrm{P2P}$ are the FAIL_TO_PASS and PASS_TO_PASS lists from above, $\bigwedge$ means "holds for every one of them", and $\text{pass}(t)$ is whether test $t$ passed. The formula says, in words: one point only if every test that should be fixed passes and not a single test that should keep working fails — otherwise zero.

As a verdict, that binary is honest: a fix that is 80% of the way there is, to the user, not a fix. As an instrument for development it is blunt, because "so close" and "died during environment setup" both score zero. That is the motive for partial credit, which we come to below.