Paper Walkthrough: Qwen-UI-Agent — How Alibaba Built a GUI Agent That Works on Real Phones and PCs

A guided tour of Alibaba MAI-UI team's technical report on Qwen-UI-Agent: a foundation GUI agent trained on 100+ physical phones, a hybrid GUI+CLI action space, and online RL over 100+ turn trajectories, reaching 92.2% on a real-device benchmark.

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Why AI That Operates Your Phone Keeps Failing on Real Roads

Picture someone who aced every driving-school test, then freezes in real traffic. That is roughly where GUI agents — AI systems that complete tasks by looking at a screen and tapping or clicking on your behalf — stand today. Research simulators are the driving school: app states reset on demand, no ads appear. Real phones are the road: pop-ups land on top of your target button, logins expire mid-task, CAPTCHAs block the way.

The paper we are reading is the Qwen-UI-Agent technical report, published by Alibaba's MAI-UI team in July 2026, and it attacks this simulator-to-road gap head on. One model covers mobile, computer use, browser use, and DeepSearch (API-based web research). It reports 92.2% on MobileWorld-Real, a benchmark run on live Android phones, ahead of Claude Opus 4.8, GPT-5.6 Sol, and Gemini 3.1 Pro (§3.3) — with a 27B-parameter model, far smaller than those systems.

The Design Philosophy: Six Transitions (§1)

The paper opens by naming six transitions next-generation GUI agents must make: (1) from simulators to real-device execution; (2) from isolated domains to cross-platform workflows; (3) from GUI-only actions to hybrid GUI+CLI and batched actions; (4) from short tasks to reliable long-horizon completion; (5) from human-intensive pipelines to AutoResearch-style development, where agents drive data construction themselves; (6) from reactive execution to proactive service initiation. The rest of this article traces how these six ideas are baked into each stage: formulation → environments → data → training → verification.

Formulation: Turning One "Step" of an Agent into Math (§2.1)

A task is the pair of an instruction $I$ and a set of available environments $\mathcal{E}_\tau$. At each step $t$ the agent receives a three-channel observation $o_t = (o_t^{\mathrm{GUI}}, o_t^{\mathrm{CLI}}, o_t^{\mathrm{API}})$ — a screenshot, command-line output, and structured API responses. Eyes on the screen, hands on a shell. The model $\pi_\theta$ produces:

Read in words, the right side is everything the agent is handed and the left side is everything it gives back. Hand it the instruction $I$ (the goal, in plain English), the current observation $o_t$ (this moment's screenshot, shell output and API response), and the history $h_t$ (everything it has already seen and done on this task), and it returns a short piece of thinking $r_t$ — the reasoning it writes to itself — together with what to actually do next, $\mathbf{a}_t$. The key detail is that $\mathbf{a}_t$ need not be a single action: it is defined as $\mathbf{a}_t = (a_t^{(1)}, \ldots, a_t^{(K_t)})$, so one model turn can emit a whole batch of actions. A routine like "click the search box → type → press Enter" needs no fresh screenshot between steps, so it collapses into one turn.

Beyond GUI operations (click, drag, type…), the action space includes cli_command for running bash directly, api_call for external services, and ask_user, which pauses for missing information or explicit confirmation before sensitive operations like payments (§2.1.2). A policy, at heart, is a probability distribution over the available actions — and how sharp that distribution is shapes the agent's character. The widget below lets you feel this.

As pseudocode, the core agent loop is compact:

h = []                                  # interaction history
while True:
    o = env.observe()                   # screenshot / CLI output / API response
    r, actions = model(I, o, h)         # one turn: reasoning + action batch
    for a in actions:                   # execute batched actions in order
        env.execute(a)                  # click / type / cli_command ...
    h.append((o, r, actions))
    if actions[-1].type == "terminate":
        break

The Environment Sets the Ceiling (§2.2)

"The environment defines the capability boundary of an agent," the authors write. On the sandbox side: mobile (the MobileWorld environment rebuilt on redroid, a containerized Android), computer use (OSWorld's Ubuntu VMs extended with direct bash execution), browser (Playwright + Chromium), and DeepSearch (Serper for ranked search, Jina Reader for page content) — with up to 10,000 isolated sandboxes running in parallel (§2.2.1).

The real-device side is this paper's signature: a runtime of over 100 physical phones and 150+ apps, used for both training and evaluation (§2.2.2). Real phones break, so the system adds a health-aware scheduler that tracks devices, apps, accounts, and network links and blacklists unhealthy ones, plus virtual displays that let one phone host several concurrent app sessions (roughly 20× rollout throughput across the cluster).