terminal-bench

Definition

A benchmark that places an agent in a command-line environment and scores whether it actually completed the task, judged by the end state of a sandboxed container rather than by the text it produced. That makes it a test of tool use and error recovery rather than of writing: the agent must inspect, act, read errors, and adjust. Difficulty comes from length — long command sequences fail if any single step goes wrong unnoticed.

Explainers using this term

• Paper walkthrough: StateM — 95.3% on Terminal-Bench 2.1 and a USD 15 run, without touching a single weightStateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling