osworld

Definition

A benchmark that evaluates agents inside real operating systems rather than a simulated stand-in. Tasks run in a virtual machine against genuine applications — spreadsheets, browsers, editors, system settings — and success is decided by inspecting the final machine state, not by string-matching an answer. That execution-based grading is what makes it a demanding test of GUI-driven computer-use agents, where long action sequences fail silently.