Paper walkthrough: ASI-Bench — peeling away human guidance to measure what AI can do alone

ASI-Bench keeps the research goal, data and grading fixed while stripping away human methodological guidance one layer at a time. Average scores fall 50.91 → 29.10 → 26.62, and the place where things break is not method selection.

ASI-Bench: At the Dawn of Artificial Superintelligence

Taking the recipe away, one page at a time

Suppose you want to measure how well someone cooks. There is more than one way to do it.

1. Hand them a recipe with exact grams and timings, and have them follow it
2. Say only "make a beef stew" — the dish, nothing else
3. Show them what is in the fridge and say "dinner, one plate"
4. Do the same as 3, but slip in a few ingredients that today's dish does not need

All four look like a cooking test, but they measure different things. The first measures execution. The third is where it starts to matter whether the cook can decide what to make. The fourth asks whether they can ignore a distraction.

ASI-Bench (arXiv:2608.17271) does this for scientific research. The research objective, the data and the grading criteria stay fixed, and only the methodological guidance a human provides is withdrawn step by step, to see how far an AI agent can get on its own (§2.2). In the paper's terms: B1 gives the complete guidance, B2 gives only the method, B3 gives only the research goal and the data, and B4 is B3 with irrelevant information mixed in.

Why another benchmark

The paper's framing is blunt. Most of what today's AI can do rests on learning, compressing and applying humanity's existing knowledge. So existing evaluations lean one of two ways: either the answer is already known, or humans have specified the procedure and the system merely completes it (§1).

Table 1 lines up the existing benchmarks and shows that each covers only part of cross-domain generality, methodological autonomy and end-to-end execution (§2.1). Humanity's Last Exam probes frontier knowledge across disciplines; SciCode targets research-level scientific coding; Terminal-Bench measures long-horizon tool use; PaperBench measures replication; MLE-Bench covers machine-learning engineering; RE-Bench covers open-ended AI R&D; DiscoveryBench covers hypothesis-driven discovery. All useful — but none of them, the paper argues, holds the objective and the grading fixed while moving only the amount of guidance.

B1 to B4: a gradient of guidance

The worked example the paper gives is a nonlinear two-dimensional dynamical system (§2.2, detailed in Appendix D).

• B1: the governing PDE, the numerical formulation and the solver procedure are all spelled out. The agent essentially implements and runs the prescribed approach
• B2: the procedure disappears. Only method-level information survives — the class of PDE and suitable numerical approaches — and the agent has to turn that into working code
• B3: even the method is gone. The agent gets the observed spatio-temporal data, the scientific objective and the required outputs. It must infer the underlying model, pick a numerical method, implement it and validate the result
• B4: B3 plus information that is factually correct but irrelevant, to see whether the agent holds its research direction

Responsibility shifts from human to AI within the same project. The contribution guidelines are explicit that B3 and B4 must never leak the intended algorithm or solver (Appendix C).

Another way to put it: the amount of guidance is how narrowly the next move has already been chosen for you. The figure below sharpens or flattens a distribution with a softmax temperature — read B1 as low temperature (nearly a single path) and B3 as high temperature (the options fan out). This is an analogy to build intuition, not a claim the paper makes.

How it was built

The scale is part of the argument. More than 40 experts and over 31,000 human-hours went into 60 project-level research tasks spanning 11 scientific domains (§2.2): mathematics, physics, chemistry, biology, astronomy, materials science, earth science, medicine and biostatistics, computer science, robotics and electrical engineering.

It was not a single pass. Construction started from over 1,300 candidate research ideas and went through five review rounds, more than 1,100 review assignments and over 2,000 task revisions. Reviewers explicitly looked for information leakage and for shortcuts that would earn a high score without actually solving the task. Every retained task was then executed end-to-end in an isolated sandbox; more than 1,500 sandbox runs were performed during development. Tasks with unresolved scientific errors, unstable execution or misaligned evaluation were revised or dropped (§2.2).

The tasks themselves are heavy: completing all 60 involves more than 2,600 interaction turns and 2,400 execution steps, spanning over 35 hours of agent execution.