A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure

What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

A ruler doesn't measure ability. It measures its own markings

If someone runs the 50-metre dash in 6.5 seconds, you know they are fast. You do not know whether they can score goals. The dash measures sprinting in a straight line, not football.

Agent benchmarks work the same way. "70% on SWE-bench" invites you to read "can do 70% of software work," but nothing supports that reading. What was measured is the pass rate on that benchmark, under that benchmark's grading rule — nothing more.

That does not make benchmarks useless. Without a ruler, nobody can tell whether a model got better. The job is not to throw the ruler away but to know exactly what it marks, and then translate its reading into your own units. That translation is what this article is about.

Three rulers

Here are the three most cited. All three share something important: they do not grade whether the model said the right thing. They grade whether acting on an environment produced the right result. What differs is which environment, and what counts as right.

SWE-bench — does the test suite go green?

SWE-bench (Jimenez et al., 2023) is built from 2,294 tasks harvested from real Python repositories on GitHub. Each task pairs an issue body with the repository state just before that issue was fixed. The agent edits code and emits a patch.

Grading is not human inspection. The tests attached to the pull request that actually closed the issue are split into two groups:

• FAIL_TO_PASS — must fail before the patch and pass after it (did the bug get fixed?)
• PASS_TO_PASS — must pass both before and after (did anything else break?)

A task counts as resolved only when both groups are green. So SWE-bench measures one thing precisely: can you write a diff that turns a specific prepared test suite green?

Two derived sets exist: Lite (300 tasks, cheaper to run) and Verified (500 tasks, human-validated). The very existence of Verified is informative — it was built because the original pool contained issues whose text under-specifies the fix, and tests that are too strict or too loose to grade fairly.

GAIA — can you research your way to one short answer?

GAIA (Mialon et al., 2023) targets general assistants: 466 questions, of which 165 form a validation set with published answers and 300 form a test set whose answers are withheld and scored through a leaderboard.

Two design choices define it. First, the questions are deliberately easy for humans and hard for models — reaching the answer requires stringing together web search, reading PDFs or tables, interpreting images, and multi-step arithmetic, across three difficulty levels. Second, answers are short strings graded by exact match. "12" or "Paris." No judge model, no human rater, no rubric.

That buys reproducibility at the cost of expressiveness. An agent can do the research correctly and still be marked wrong for formatting the answer differently.

OSWorld — can you drive a computer into the right state?

OSWorld (Xie et al., 2024) uses entire virtual machines running real operating systems (primarily Ubuntu, with Windows and macOS as well). Its 369 tasks include workflows that span several applications: file management, browsers, office suites, the terminal.

The grading is the interesting part. Each task ships with a setup script that builds the initial state and an executable validator that inspects the final state. "Sort these columns in this spreadsheet" is graded by opening the file afterwards and checking its contents. The route is not graded; only the destination is.

That makes OSWorld the closest of the three to production feel — and also the most fragile. Network conditions, application versions, and unexpected dialogs all move the number.

The grading rule sets a ceiling on what can be measured

All three scores have the same shape:

$$\text{score} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\!\left[V_i(s_i^{\text{final}}) = 1\right]$$

$N$ is the number of tasks, $s_i^{\text{final}}$ is the environment state after the agent stops working, $V_i$ is that task's dedicated validator (run the tests, compare the string, inspect the file), and $\mathbf{1}[\cdot]$ returns 1 when its contents are true and 0 otherwise. The whole line in words: run every task, let each task's own validator look at the state you left behind, and report the fraction that got a tick. Nothing else.

Written this way, the limits fall out of the formula. Anything $V_i$ does not look at contributes exactly zero to the score. How many API calls it took, how many dollars it burned, whether it deleted a file it should not have touched, how often it succeeds if you run it ten times — all outside $V_i$.

Attempt count is the other half of the translation. Agent runs are stochastic, so if you run a task $n$ times and succeed $c$ times, pass@$k$ estimates the chance that drawing $k$ of those runs yields at least one success:

The formula in words: given that this task came out right $c$ times in $n$ recorded runs, how often would a handful of $k$ runs contain at least one success? $\binom{a}{b}$ is the number of ways to choose $b$ items out of $a$. The fraction is the probability that all $k$ draws are failures, so subtracting it from 1 gives the probability of at least one success.

This is a rich source of misreadings. pass@1 (get it right first try) and pass@5 (one hit out of five is enough) can differ enormously for the same model. Comparing two published numbers without checking whether each is pass@1, best-of-$n$, or majority vote makes the comparison meaningless.

What isn't being measured

Of everything sitting outside the grading rule, these matter most in practice.

Cost. The same 70% means something very different at $0.30 per task than at $15 per task. Most benchmarks do not require cost reporting, so the leaderboard cannot tell you.

Scaffold contribution. A SWE-bench number is not "the model." It is model + retrieval tooling + edit tooling + retry loop. The same model can move by tens of points depending on the harness around it. Believing you are comparing models when you are actually comparing someone's scaffold is a routine mistake.

Ambiguity of the request. Real work starts with "login has been weird lately." A SWE-bench issue has already been curated to the point where at least the target tests are determined.

Side effects and safety. An OSWorld validator checks that the intended state exists. It is not designed to penalize "and also overwrote an unrelated file along the way." In production, where some actions cannot be undone, that omission is the whole ballgame.

Variance. On a 300- or 369-task set, a two-point gap can vanish into noise. One task is roughly 0.3 points, so a handful of coin flips reorders the leaderboard. A score difference published without confidence intervals or multiple seeds should not be read as a capability difference.

Four kinds of contamination

The other reason a high score need not mean high capability is contamination. The word covers at least four distinct failures, which get conflated constantly.

1. Training-data contamination. The benchmark's questions and answers sit in the model's pretraining corpus. SWE-bench is built from public GitHub repositories, so the fixing commits and the issue discussions are, in principle, crawlable. The score alone cannot distinguish "solved it" from "remembered it." The usual diagnostic is the cutoff comparison: if performance on tasks created after the model's data collection date is clearly worse than on earlier tasks, that is evidence of contamination.

2. Solution leakage inside the environment. Here the answer is not in the weights — it is sitting in the material handed to the agent. A commenter wrote "just change line 82 of this file" in the issue body; another branch, or the test file itself, contains the post-fix expected values. An agent can grep its way to a passing patch without reasoning at all. Much of the human validation behind SWE-bench Verified was exactly this filtering work.

3. Weak validators. The tests are lax enough that a non-substantive patch passes. If the suite only asserts f(1) == 2, then return 2 is a passing implementation. Execution-based grading removes rater subjectivity, but it comes with a permanent trade: it only certifies correctness within the space the tests describe.

4. Leaderboard overfitting. This one creeps. You submit to the test set, look at which tasks failed, adjust prompts and scaffolding, submit again. Each individual step is legitimate tuning, yet repeated enough times, the test set's answers leak into your system through your own head. You can overfit a test set having never once touched the training data.

The intuition for the fourth kind is just plain overfitting. In the figure below, raising the polynomial degree hugs the training points ever more tightly while test error turns around and climbs. On a leaderboard, the test set is sitting in the "training data" seat.

To investigate the first kind, the first practical move is to look for overlap between the benchmark text and the corpus. The naive version counts matching n-grams:

def contaminated(question, corpus_docs, n=13, thresh=0.5):
    grams = {question[i:i+n] for i in range(len(question) - n + 1)}
    for doc in corpus_docs:                       # training corpus side
        hit = sum(1 for g in grams if g in doc)
        if hit / max(len(grams), 1) > thresh:     # overlap above threshold
            return True
    return False

String matching misses paraphrases and translations, so teams pair it with embedding-based nearest-neighbour search over the corpus. The figure below makes the catch visible: drag the query and the top results reshuffle depending on whether you rank by dot product, cosine, or L2. Contamination detection inherits exactly that sensitivity — your verdict depends on the distance you picked and where you set the threshold.

Translating a published score into your units

When a number lands in front of you, walk down this list.

1. Which subset? SWE-bench Full, Lite, and Verified sit at different levels. Never compare without checking the name.
2. What scaffold? Not just the model — the agent framework, the tools, the step limit, the retry budget.
3. pass@what? One shot, best of several, or majority vote.
4. Cost and wall-clock per task. If unreported, that means unknown, not cheap.
5. Cutoff relationship. Were these tasks public before the model's data was collected?
6. Distance from your distribution. SWE-bench is twelve Python repositories; GAIA is the English-language web; OSWorld is Ubuntu-centric. If you run a different language, an internal system, or a different OS, that gap is pure extrapolation.

Then ask the question that does the most work: does this benchmark's validator catch the failures I actually fear? If your production nightmare is "looks fine but corrupts data on the error path," a rule that only requires FAIL_TO_PASS to go green never looked at that failure once. In which case the score is context, not evidence.

How this shows up on the job

When an ML engineer or eval owner is selecting a model or an off-the-shelf agent product. Use public scores for first-pass screening only — eliminating obviously weak candidates. Make the final call on an internal benchmark of 30–100 tasks drawn from your own work. The fastest way to build one is to copy the structure of the public benchmarks: for a SWE-bench-shaped suite, curate per-task "must start failing" and "must not break" tests; for an OSWorld-shaped one, write a setup script and a final-state validator. Inspect the destination, not the route — hold that line and you can rewrite the agent without rewriting the evaluator.

Names you will actually touch. On the SWE-bench side: FAIL_TO_PASS / PASS_TO_PASS, SWE-bench Verified, and a harness that spins one Docker container per task. On the GAIA side: the three Levels, exact-match scoring, and the withheld test answers. On the OSWorld side: VM snapshots, the setup configuration, and the evaluator function built from getters and metrics. Across all of them: pass@$k$, max steps, tokens and dollars per task.

Traps that turn into incidents.

• Tuning prompts against your internal test set every day. That is kind four, in-house. Within two or three weeks the internal number drifts away from how the system actually feels. The fix is to split into a development set and a sealed set, and touch the sealed set once a month. It moves less, which is boring, and which is the evidence that it is working.
• Comparing public scores with mismatched scaffolds. Model A behind a full-featured framework, model B as a bare single call, presented in one table. Extremely common.
• Running once and concluding. Agent runs have high variance, and OSWorld-style environments drift for reasons unrelated to the agent. Run at least three times, ideally five, and look at both the mean and the worst case. A mean-only view hides the agent that occasionally does something destructive.
• Quoting decimals off a tiny internal suite. At 50 tasks, one task is two points. "2.1 points better" may be one flipped coin.
• Putting production credentials in the eval environment. Agents do unplanned things. Make disposable VMs, restricted networking, and read-only mounts the default.

How this shows up in an interview or a design review. "This benchmark score is high — where might it fail to transfer to our environment?" Structure the answer in four parts and you will not leave gaps: (1) failures the validator never inspects (cost, side effects, variance); (2) distribution gap (language, OS, internal systems); (3) contamination and leakage (cutoff relationship, answers present in the environment); (4) scaffold contribution.

Summary

• SWE-bench, GAIA, and OSWorld respectively measure "turn a test suite green," "produce one short answer that matches exactly," and "reach a target machine state." The shared form is only "the fraction of tasks a validator ticked."
• Everything outside the validator — cost, side effects, variance, scaffold contribution — is invisible in the score.
• Contamination comes in four kinds: training-data contamination, solution leakage inside the environment, weak validators, and leaderboard overfitting. The fourth happens even if you never touch training data.
• Public scores are for screening. Decide on an internal benchmark built from your own tasks, split into a development set and a sealed set.

For how agents are constructed in the first place, see LLM Agents: The Basics; for the mechanics of building validators, Agent Evaluation; and for testing contamination empirically, Data Leakage: Verified by Experiment.