terminal-bench

Definition

A benchmark that places an agent in a command-line environment and scores whether it actually completed the task, judged by the end state of a sandboxed container rather than by the text it produced. That makes it a test of tool use and error recovery rather than of writing: the agent must inspect, act, read errors, and adjust. Difficulty comes from length — long command sequences fail if any single step goes wrong unnoticed.