SafeActBench explained (arXiv:2610.07753) — tool agents fail before execution

A correct outcome does not mean the action was supported by evidence gathered beforehand. Across 656 SafeActBench cases and ten model-harness configurations, static action judgment scores 83.7-97.7 while interactive execution falls to 47.3-72.6 — and most failures start before execution.

From Evidence to Action: How Tool-Using Agents Fail

A correct outcome is not evidence that the agent had evidence

The paper this article walks through is "From Evidence to Action: How Tool-Using Agents Fail" (Hongzhan Lin, Shidong Cao, Ziyang Luo, Wenhao Chai, Mong-Li Lee and Wynne Hsu; arXiv:2610.07753, 6 October 2026).

Tool-using agents change external state: they send the mail, rewrite the config, release the payment. The awkward part is that a correct outcome does not guarantee the action was supported by evidence established beforehand. A lucky action and an investigated action look identical if you only grade the result.

This paper measures where that evidence-to-action chain breaks.

What they built

SafeActBench: per the abstract, 656 cases across six operational domains and five protocols that progress from static action judgment and investigated non-action to single- and multi-action workflows.

Two pieces carry the measurement:

• a provenance-bound Evidence Ledger — what information was established, and when
• a deterministic trajectory evaluator — when actions occurred, and whether downstream dependencies were satisfied

So the unit of grading is not "did it get the answer" but what did it know before it acted.

Measured: strong on paper, much weaker with tools in hand

Ten model–harness configurations. The harnesses are Claude Code, Codex, DeepSeek Harness (DSH), Qwen Code and ZCode, plus a shared ReAct harness implemented in Inspect AI (Inspect), which allows harness-controlled comparisons of the same model.

Columns: overall (ECS, P-M, D-M — protocol- and domain-macro averages) and per protocol (Legacy, V0–V3).

One reading. Legacy — static action judgment — runs 83.7 to 97.7, while the interactive protocols fall to between 47.3 and 72.6. Same models. Judge an action on paper and they are right more than nine times in ten; hand them the tools and they land near half. In the abstract's words: strong static action assessment can coexist with much weaker interactive execution.

The harness matters too. GPT-5.6 scores ECS 65.4 under Codex and 61.0 under the shared Inspect harness; Qwen3.8 scores 66.0 under Qwen Code and 63.9 under Inspect. Change the container, not the model, and the number moves — the same direction as the HarnessDev walkthrough.

The failures start before execution

This is the finding that changes how you build. Three points from the abstract.

1. Failures often begin before execution. Agents stop with incomplete investigation, or act before required evidence is established.
2. Once required evidence is obtained, single-action execution is usually reliable.
3. Multi-action workflows additionally expose unresolved prerequisites and incomplete execution.

The order matters. It is not only that information was missing. Failures also arise from how agents use established evidence when deciding and executing — that is the paper's closing claim.

What changes for implementers

• Stop accepting "the output was right" as a pass. Without recording when the evidence arrived, a lucky action grades the same as an investigated one.
• Count the way it stops as a failure too. Backing off with "I can't tell" while information is still obtainable is its own failure mode: when relevant information remains obtainable, missing or unresolved evidence should prompt further investigation rather than premature refusal.
• Multi-action is not single-action repeated. Resolving prerequisites and finishing the job are separate requirements.

In practice

Engineers wiring tool-using agents into real work, people building agent evaluation suites, and whoever operates those agents in production will use this paper's lens in three places.

1. Log when the agent knew, not just what it did. An evaluation that grades only the end state — was the ticket closed, does the config hold the expected value — passes lucky actions along with investigated ones. At a minimum, keep the tool-call trajectory: which tool, which arguments, in what order, and what came back. Then make it possible to check, at the moment of each write action, whether the reads that decision depended on had already happened. In Inspect AI, the eval log keeps each sample's messages and tool calls, and inspect view lets you walk the trajectory. Writing your scorer against the trajectory instead of the final output is a small, home-grown version of the paper's Evidence Ledger and deterministic trajectory evaluator.

2. Put the gate in front of the write tools. If most failures are "acted before the evidence was in", the place to stop the agent is just before execution. Claude Code's PreToolUse hooks and its tool permission settings, and Codex's --sandbox modes (read-only / workspace-write) and approval policy, are exactly the knobs for that boundary. A sensible starting split: reads run freely; anything that changes external state — sending, deleting, paying, touching production config — needs a confirmation. But a gate does not judge whether the evidence is sufficient. It only creates a moment where a human approves. Unless the approval prompt shows what the agent checked before proposing the action, the approver ends up judging by how the result looks, too.

3. Do not count "I can't tell" as a success. A refusal that errs on the safe side looks harmless, but this paper scores stopping while information is still obtainable as a failure. In operations, do not just track a refusal rate. When an agent stops, check whether reads it never made were still available — an unopened file, a lookup API it never called. If so, the fix belongs in the prompt or the tool descriptions, pushing the agent to keep investigating.

Pitfalls that bite

• Estimating workflows from single-action success. In the paper, single actions are usually reliable once the evidence is in. Multi-action workflows add unresolved prerequisites and runs that stop partway. Every step can pass its own test while the workflow still fails; keep a separate end-to-end test.
• Comparing across mixed harnesses. As the table shows, the same GPT-5.6 scores ECS 65.4 under Codex and 61.0 under Inspect. To compare models, fix the harness; to compare harnesses, fix the model.
• Claiming safety from static judgment tests. A model that scores 83.7–97.7 on "should this action be taken?" questions on paper can fall to 47.3–72.6 once it holds the tools. A judgment-test score is not a substitute for execution safety.

Questions you will get in design reviews and interviews

• "Of the cases where this agent got the right result, how many did it reach after checking the evidence?" The key is recognising that end-state grading cannot answer this. You need saved trajectories and a check that the required reads had happened by the time of each write action — only then can you count.
• "The refusal rate went down. Is that an improvement?" It means opposite things depending on whether unnecessary refusals fell or the agent started acting on thin evidence. Unless the two are counted separately, you cannot tell.

What this article does not establish

• It does not say what V0–V3 are. The fetched source text does not include the body of §4.2, so nothing beyond the column labels can be stated. The progression comes from the abstract (static judgment → investigated non-action → single action → multi-action).
• The table shows 8 of the 10 configurations; the ZCode row was not recoverable from the fetched text.
• Per-domain case counts are omitted: the mapping between the six domain names and their counts could not be fixed from the extraction.

Every figure here is checked verbatim against the fetched source (arXiv:2610.07753).