Paper Deep Dive: From Evidence to Action: How Tool-Using Agents Fail — Reading the Failure Diagnostics and Interventions

SafeActBench splits tool-using agent failures into a chain of investigate, act, and hand off. This deep dive reads the paper through its appendices: how each failure diagnostic is defined, what agents do when decisive evidence is removed, and whether structuring evidence actually helps.

From Evidence to Action: How Tool-Using Agents Fail

A correct outcome is not enough

The paper is "From Evidence to Action: How Tool-Using Agents Fail" (Hongzhan Lin et al., arXiv:2610.07753, 2026-10-06). If you only want the big picture, there is a free overview. This piece goes through the main text and appendices to see exactly how failures were split up and measured.

Here is what the abstract claims. Tool-using agents change external state, yet a correct outcome does not mean the action was backed by evidence the agent had established beforehand. The authors trace where this evidence-to-action chain breaks as agents move from deciding whether to act, to executing a single action, to running workflows with dependencies, across ten model–harness configurations. Strong static judgment turns out to coexist with much weaker interactive execution. Failures often start before execution: agents stop before finishing their investigation, or act before the required evidence is in hand. Once that evidence is established, single actions are usually reliable, while multi-action workflows add unresolved prerequisites and unfinished execution. To measure this, the authors build SafeActBench (656 cases, six domains, five protocols), a provenance-bound Evidence Ledger, and a deterministic trajectory evaluator. The conclusion: failures come not only from missing information but from how agents use the evidence they already have.

An analogy: the new hire processing refunds

The paper opens with this example (§1). You ask a new hire to "refund the duplicate charge and let the customer know." The right sequence is to identify the charge, confirm it really is a duplicate and that the refund conditions hold, issue the refund, and put the returned refund ID in the message.

Now suppose the new hire looks at a different charge, C1, sees the same amount, and refunds C2. The refund succeeds and the amount is right. But nothing was ever checked about C2, so the action counts as unsupported (§3.1).

Three links in the chain

• Evidence is bound to its entity: another charge's amount does not support an action on C2, even when the values match (§3.1)
• Not acting also requires investigation: when the right answer is to do nothing, the agent still has to look into what matters first. The paper calls this investigated non-action (§3.2)
• Results flow forward: if a refund returns refund_id=R42, the notification must use that value, and in the right order (§3.3)

The 656 cases fall into five protocols (Table 1).