Evaluating Agents — How Benchmarks and Harnesses Are Built

An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

What does "50% on SWE-bench" actually claim?

Suppose a launch deck says "SWE-bench Verified: 50%." Read plainly, that means the system can fix half of real GitHub bugs. But the number is not a property of the model. It is produced by four things together — the model, the scaffolding (the harness), the working environment, and the grading rule — and three of them never appear in the bar chart.

Think of hiring. Someone who aces a multiple-choice written test and someone who is handed a repository and told "please fix this bug" are being measured on different things. The second job rewards knowing how to run the test suite, not breaking a neighbouring feature while fixing yours, and backing out when you are stuck. Agent evaluation is trying to measure that second thing, which is exactly why the measurement itself becomes a design problem.

The three questions you ask of any benchmark — what was measured, how was it graded, and did the questions leak — all still apply. What follows is what gets added on top.

Three ways this differs from a quiz

1. There is an environment. The input to a multiple-choice question is a string. The input to an agent is a repository, a shell, a filesystem. It is not just larger — the agent can rewrite its own input. In the extreme, it can delete the failing test and report that everything passes. Whoever grades has to control both the initial and the final state of that environment.

2. There is a horizon. A single generation is one roll of the dice; an agent takes dozens of steps. Write $p$ for the per-step probability of not derailing and $n$ for the number of steps a task needs:

Here $p_i$ is the chance that step $i$ does not derail, $n$ is how many steps the task takes, and $P_{\text{success}}$ is the chance of finishing. Put in words: the end-to-end success rate is the per-step survival rates multiplied together — an obvious statement.

It stops being obvious once you put numbers in. At 99% per step over 50 steps, $0.99^{50}\approx0.61$; at 95%, $0.95^{50}\approx0.077$. Four points per step becomes almost an 8× gap end to end. Turned around: staring at the end-to-end number alone will never tell you which step is short.

3. There are side effects. Grading a quiz ends with checking an answer key. An agent writes to the world: it deletes files, runs commands, calls APIs. "The eval environment held production credentials" is an accident that happens before grading is even a question. An agent evaluation environment has to be built as a disposable box you are willing to destroy.

Opening up one SWE-bench instance

SWE-bench (Jimenez et al., 2023) is built mechanically from real GitHub issues and the commits that actually closed them. It contains 2,294 tasks drawn from 12 Python repositories, and one instance is made of these parts:

{
  "repo": "django/django",
  "base_commit": "…",        // state just before the fix; work starts here
  "problem_statement": "…",  // the issue text — in principle the only input the model gets
  "patch": "…",              // the fix a human actually wrote (reference; never shown)
  "test_patch": "…",         // the tests that landed alongside that fix
  "FAIL_TO_PASS": ["…"],     // tests that fail before and must pass after
  "PASS_TO_PASS": ["…"]      // tests that must keep passing either way
}

The design decision that matters is that correctness is not scored by similarity to the human patch. There are many valid ways to fix a bug, so string comparison would mark correct fixes wrong. What replaces it is execution: FAIL_TO_PASS carries "it is fixed" and PASS_TO_PASS carries "nothing else broke." That second half is what rejects a patch that works and takes a neighbouring feature down with it.

Learn the subset names too. Lite (300 instances) is a re-selected subset that is cheaper to run; Verified (500 instances) was checked instance by instance by people, dropping issues whose text under-specifies the fix and tests so strict that a correct fix still fails. Before comparing numbers, check which set produced them — with a different population, the same "50%" is a different quantity.

"resolved" is defined as a conjunction

SWE-bench's headline metric, the resolve rate, is 0 or 1 per instance:

$\mathrm{F2P}$ and $\mathrm{P2P}$ are the FAIL_TO_PASS and PASS_TO_PASS lists from above, $\bigwedge$ means "holds for every one of them", and $\text{pass}(t)$ is whether test $t$ passed. The formula says, in words: one point only if every test that should be fixed passes and not a single test that should keep working fails — otherwise zero.

As a verdict, that binary is honest: a fix that is 80% of the way there is, to the user, not a fix. As an instrument for development it is blunt, because "so close" and "died during environment setup" both score zero. That is the motive for partial credit, which we come to below.

The harness decides half the score

The same model moves several points depending on what surrounds it. At least four knobs matter.

• How information is supplied: just the issue text, or retrieved candidate files, or — in the oracle setting — the files the reference patch touches, handed over up front. The last one measures a ceiling, not a number you will see in production.
• The shape of the tools: SWE-agent (Yang et al., 2024) showed that resolve rates move when you change only the agent-facing interface — showing files through a fixed-size window, returning syntax-check results after every edit. Keep the model, change the tools, and the score changes.
• The budget: maximum steps, token ceiling, command timeouts, number of attempts.
• Grading details: what happens when the patch fails to apply, how long a test run may take, how flaky tests are treated.

So the honest phrasing is not "model A gets 50%" but "model A with harness B under settings C gets 50%." When comparing, move one thing at a time — ordinary experimental hygiene, applied to evaluation.

There is also a dynamic specific to public benchmarks. Tune prompts and tools a few dozen times while watching 300 or 500 instances, and you accumulate work that helps on that set and does nothing for your own repository. Each individual adjustment is a legitimate improvement; nobody acted in bad faith. The result is still overfitting.

It is the same structure as overfitting to a validation set in Overfitting and Evaluation Design, so the remedy is the same: the set you tune on and the set you decide on have to be different sets.

The environment itself is holding the answer

Contamination in agent evaluation is not only about training data. Sometimes the answer is sitting inside the working environment. There are four main routes.

1. Git history: even after checking out base_commit, if later commits remain in the clone's object store, git log --all reads out the reference patch. Truncating history and removing remotes and tags is the minimum.
2. The network: search the issue number and you find the pull request that closed it. If you allow online lookup, the score quietly turns into a measurement of how good the agent is at searching. Disable it by default (no network in the container), and if you do allow it, publish that fact next to the number.
3. The grading tests: leave test_patch in place from the start and the agent can read the expected function names and return values. So grading tests go in after the agent has finished, and immediately before grading you restore the test files from the pristine copy and then apply the test patch. That mechanically voids any run where the agent edited the tests into passing.
4. Pretraining: fix commits in public repositories exist on the web and can end up in training data. The countermeasure is rebuilding the set from issues after the cutoff — the same move, familiar from evaluation generally, of writing fresh problems to the same recipe and measuring again.

What tips you off is memorisation behaviour: paraphrase the issue text and it suddenly cannot solve it, rename a variable and it fails, the patch is right but the explanation does not match it. Routes 1 and 2 are yours to close, so check those first.

How to design partial credit

The binary is correct, but during development it is high-variance and shows no progress. If you add partial credit, this is the shape it should take:

$\mathbb{1}[\cdot]$ is 1 when the condition inside holds and 0 otherwise, $t_i$ ranges over the FAIL_TO_PASS tests, and $w_i$ is a weight. Put in words: make "you broke nothing" the entry condition, then score the fraction of the target tests you fixed.

Four principles hold this together.

• Keep the gate binary. Give partial credit for regressions and "fixed feature A, broke feature B" becomes a scoring gain.
• Put partial credit on intermediate milestones — environment set up, reproduction test written, right file reached — things you can observe. Never score a quantity unrelated to the outcome, such as string similarity to the reference patch; that optimises for writing patches that merely look alike.
• Always publish the binary resolved alongside it. Partial credit is an internal instrument, not an external result. Publishing both is also how you notice partial credit rising while resolved stays flat.
• Anything published gets optimised. The moment you reveal which test is worth what, implementations appear that satisfy exactly those tests. You need held-out tests and periodic rotation.

One run is not enough

Agent runs do not reproduce, even at temperature 0: batching on the inference side, response times of external commands, environment state that depends on the clock. Different trajectories from the same input are the norm, so a gap seen in a single run may be noise rather than skill.

• pass@k (succeeded at least once in k tries) is a product number only when something can pick the correct one out of the k. Without a selector, it measures an upper bound.
• pass^k (succeeded in all k tries) measures reliability; τ-bench reports this shape. For an agent that finishes half the time, the naive chance of doing the same job right five times running is $0.5^5 \approx 0.03$ — and that is the number users feel.
• Report sampling error. On a 500-instance benchmark, a three-point gap is usually within noise. The arithmetic for "how many items before I believe this number" is in Evaluating RAG in Practice.
• Separate out flaky tests. Tests that depend on execution order or the clock will fail a correct patch. The control experiment here is running the reference patch through your own harness: if that does not come out at 100%, what is failing is the harness, not the model.
• Publish cost next to score: tokens per instance, wall-clock, step ceiling. Ten times the budget for three more points is not a win.

In code: the smallest possible grader

Turning all of the above into a scoring function directly:

def score(inst, model_patch):
    repo = checkout(inst.repo, inst.base_commit)   # disposable clone, no history, no network
    if not apply(repo, model_patch):
        return 0.0                                 # a patch that won't apply scores 0, not an exception
    restore(repo, inst.test_files)                 # throw away any edit the agent made to tests
    apply(repo, inst.test_patch)                   # grading tests go in last
    r = run_tests(repo, timeout=900)               # the cutoff is part of the grading rule too
    if any(r[t] != "pass" for t in inst.PASS_TO_PASS):
        return 0.0                                 # regression gate: break something, get nothing
    return mean(r[t] == "pass" for t in inst.FAIL_TO_PASS)

resolved = score(inst, patch) == 1.0               # this is the number you publish

The ordering carries the meaning. Putting restore before apply(test_patch) closes both leakage and tampering; putting the regression gate before the average stops partial credit from hiding a regression. Swap either one and it breaks silently.

How this is used on the job

The people who write agent evaluations are whoever decides if a coding agent goes into the company, the engineers building the agent platform, and whoever signs off on switching vendors or models. The moments are model swaps, pre-release regression checks, and answering "does this actually work on our code?"

What you touch, concretely: on the benchmark side, FAIL_TO_PASS / PASS_TO_PASS, the choice between Lite and Verified, and pinning the per-instance Docker image tag. On the harness side, maximum steps, token ceiling, test timeout, disabling the container network, temperature and seed. On the operations side, storing the full trajectory (commands and outputs) and running the reference-patch control. Throw the logs away and you will not be able to explain a single failure afterwards.

The accidents come from a familiar cast. Comparing against historical numbers right after changing the harness (they are no longer comparable). Leaving the network open and watching the score jump (you measured search). Not detecting edits to tests. Reporting a three-point gap from one run. Optimising against partial credit and growing a test-specific implementation. Counting flaky-test failures as model failures. Omitting cost.

In interviews and in real work, the question is usually the same one: "this agent gets 50% on SWE-bench — what does it get on our repository?" Break the answer into three parts. Distribution shift (SWE-bench is well-known Python OSS where a test suite already exists; internal code has different test density), contamination (fixes to those projects may already be in training data), and harness gap (published numbers come with heavily engineered scaffolding). Which makes the honest answer "we cannot know without measuring — start by turning 30–50 of our own issues into a benchmark." The fastest way to build that in-house set is to convert the failure modes from LLM Agents from Scratch directly into things you observe and record.

Summary

• An agent score is set by four things together: model, harness, environment, grading rule. Compare without pinning three of them and you have said nothing
• SWE-bench measures "it is fixed" with FAIL_TO_PASS and "nothing else broke" with PASS_TO_PASS, scoring a point only when both hold
• Contamination enters through git history, the network, the grading tests, and pretraining. The first three are yours to close
• Partial credit is safe only with a binary regression gate, milestones as the scored events, and the binary score published alongside
• One run is not enough: put pass^k, sampling error, the reference-patch control, and cost in the same place as the score