Paper Walkthrough: FrontierChallenge — Grading Scientific Work on Whether It Was Actually Delivered

A walkthrough of FrontierChallenge, a benchmark that scores whether an agent finishes a scientific workflow. Average scores hit 87.9 while full completion stayed at 20.6% — and 0% in electrochemistry, where the average was 94.9. Worse, 75.5% of failing trajectories still ended by claiming they were done.

FrontierChallenge: Evaluating Scientific Workflow Completion

Some work can't be accepted at "mostly finished"

The movers delivered 9 of your 10 boxes. That's 90%. But if the missing box holds the fridge power cord and the toolkit, your new life still doesn't start today.

Scientific work behaves the same way. Analyzing data doesn't just mean producing a number. A rerunnable analysis script, a per-item table, a group-statistics table, figures you can sanity-check, a report describing the method — only when all of it lands can the next person pick the work up. If one figure is missing, nobody can verify whether the remaining numbers are right. "90% done" is often a synonym for "not handed off."

FrontierChallenge (arXiv:2608.24979, Apodex Team, August 2026) is a benchmark built to measure exactly that gap.

The single question the paper asks

The paper narrows its question aggressively (§1). Given a specified scientific task and fixed data, can an agent independently complete the workflow from input processing to final deliverables and satisfy the complete task contract?

The authors are explicit that this is not autonomous science (§1). Setting the research agenda isn't part of the job. The objective, the inputs, and the required outputs are all fixed in advance; what's measured is only whether the specified work gets carried to the end and handed off in a mutually consistent form. That's the departure from HLE, SWE-bench, and OSWorld, whose evaluation units are a final answer, an interaction trace, or a single program (§2).

How the benchmark is built: writing a task as a contract

The authors assembled 300 end-to-end scientific workflows, released and evaluated 97 of them, and kept the remaining 203 as an internal held-out set (§3). Representative tasks require domain software such as ORCA, CP2K, LAMMPS, AmberTools, and PLUMED, and demand executable code plus evidence-grounded reports (§1).

The packaging is the interesting part. Each task is assembled from five aligned elements (§3.1): a task description, fixed inputs including data and context, the software and tools in the execution environment, an output contract listing the required deliverables, and an evaluation procedure defining completion. The unit of evaluation is not a final answer but the entire submitted artifact bundle. In the wound-healing assay of Appendix B.1, 18 bright-field microscopy images (two groups × 0/12/24 hours × three replicates) must produce an analysis script, an image-level table, a group-statistics table, two summary plots, and a methodological report.

The released 97 break down as quantum chemistry 20, molecular dynamics 16, materials characterization 22, analytical chemistry 23, life science 10, and electrochemistry/environment 6, split into 74 Hard and 23 Medium (§3.2). The paper adds a warning: these six domains are descriptive slices of the release, not probability samples, so cross-domain differences must not be read as an intrinsic ranking of disciplinary difficulty (§3.2).

Two metrics: Pass Rate and Avg. Score

Each task's Grader inspects required files, numerical results, formats, figures, code execution, and — crucially — cross-artifact consistency, returning a score $s_{mi}$ in 0–100 for configuration $m$ on task $i$. Semantic criteria are delegated to a Judge (GPT-5.6 Sol, run three times per criterion), but the Judge remains part of that task's Grader (§4.2). With $N = 97$:

$\mathbb{1}[\cdot]$ is a switch that returns 1 when the condition inside holds and 0 otherwise. In words: $f_{mi}$ is a 0/1 flag for "was this task finished completely," so Pass Rate is the fraction of raised flags and Avg. Score is the mean raw score. The odd 99.9 threshold only absorbs the numerical jitter introduced by averaging three Judge passes; it relaxes no rubric requirement (§4.2).

So the paper treats "85" and "100" not as a difference in quality but as the difference between delivered and not delivered. An 85 is the move where only the fridge is missing.

An aside: why partial credit doesn't turn into full credit

This part is not the paper's claim — it's a reading aid. If a task has $n$ required deliverables and each is satisfied independently with probability $p$, everything landing has probability $p^n$. Even at $p = 0.95$, $n = 20$ gives about 0.36. The average moves gently while the full-completion rate falls exponentially.