gui

Definition

Graphical user interface — windows, buttons, menus. In agent research the term names a specific setting: the agent acts through the same visual interface a human would use, taking screenshots as observations and clicks and keystrokes as actions, rather than calling a documented API. That framing is what makes it hard, since the agent must localize targets visually and cope with layouts that were never designed to be machine-readable.

Explainers using this term

• Paper Walkthrough: Qwen-UI-Agent — How Alibaba Built a GUI Agent That Works on Real Phones and PCsQwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents