Paper Walkthrough: UI-Venus-2 — Taking Screen-Operating Agents From Benchmarks to Real Work

A ground-up read of the UI-Venus-2 Technical Report: a GUI agent that drives phones, browsers and desktops from pixels alone. What it scales, how it verifies its own training data, and where it still falls short.

UI-Venus-2 Technical Report

The paper we are reading

The original title is "UI-Venus-2 Technical Report" (Venus Team, Ant Group; arXiv:2609.00028, published 27 August 2026). Search and cite it under that English title.

Here is the abstract in plain terms. Multimodal GUI agents look like a promising way to automate digital work, but getting from a benchmark-oriented model to something you can depend on is hard for three reasons: limited environment coverage, brittle task construction, and unreliable reward verification. The paper presents UI-Venus-2, a general-purpose foundation GUI agent that works across mobile, web and desktop through a single closed-loop reasoning–action framework. To close the gap toward deployment, the authors scale three dimensions jointly: (1) environments, expanding to 170+ multilingual mobile apps and native desktop operating systems; (2) tasks, using a deep-research pipeline that grounds generated instructions in real application functionality; and (3) verification, using trace-level and sample-level evaluators with visual keypoints and multi-model voting so RL rewards are trustworthy. They also add safety-aware mechanisms for controlling consequential actions, and release the model openly.

An analogy: training a new operator for production

Picture onboarding someone to do back-office work. Three things are on you. Give them more systems to touch — someone who has only ever used one internal tool freezes in front of a second. Give them exercises tied to functions that actually exist — "try something" teaches nothing. And grade them correctly. The last one is the most boring and the most important: if your grading is sloppy, the trainee learns to satisfy the grader rather than do the job.

That is exactly the paper's claim: environments, tasks and verification have to be scaled together, not separately, or benchmark numbers rise while real-world reliability does not (§1).

Why "a better next action" isn't enough

A GUI agent looks at the rendered screen rather than an API, and acts the way a person does — clicking, typing, scrolling. That is the appeal: you can automate applications that never exposed a usable API (§1).

The paper spells out where this stalls. First, coverage has to grow from a handful of apps to multilingual mobile ecosystems, dynamic websites and full desktop operating systems. Second, more environments means automated task construction, and a generated instruction is worthless unless it is actually executable in that app under changing interface states. Third, reinforcement learning is only as good as the verifier that supplies its reward: a coarse verifier mistakes partial progress for completion and exposes a reward signal the policy can exploit (§1). We cover that failure mode in Reward Hacking.

The base models, and the inference settings

UI-Venus-2 comes in 9B and 27B variants, initialized from the open multimodal foundation models Qwen3.5-9B and Qwen3.6-27B. Training uses a deliberate mixture of Grounding, CAPTCHA, Mobile, Web and Computer tasks, which the paper frames as complementary rather than independent: grounding supplies fine-grained spatial perception, CAPTCHA supplies controlled and verifiable interaction supervision, and Mobile/Web/Computer supply navigation experience in realistic environments (§2.1).

The inference configuration matters in practice too. For general agentic tasks, sampling temperature is 1.0, thinking mode is on, and the reasoning history is fed back into context to keep multi-step interactions coherent. For GUI grounding, thinking is disabled and temperature is 0 — if the job is to hit a coordinate, you want a direct, deterministic answer (§4.1.1). The figure below lets you feel what temperature does to a distribution.

The three-stage training pipeline

The recipe has three stages (§2.2–§2.4).

Stage I: multimodal mid-training. A large, heterogeneous mixture of synthetic and interaction data dominated by Mobile / Web / OS navigation. Collected trajectories pass through a human–discriminator collaborative verification process — human inspection plus automated trajectory-level assessment — that removes invalid, ambiguous or low-quality interactions (§2.2).

Stage II: offline reinforcement learning. For Mobile, OS and Web they build large-scale step-level RL trajectories, supervising individual steps to optimize state-aware action selection, multi-step navigation, transition consistency and execution reliability. For CAPTCHA and Grounding they instead use programmatic synthesis, embedding verified instances and grounding targets into realistic page and app backgrounds. Because the instance is generated, action-level correctness is reliable and difficulty is controllable — dense RL supervision that is hard to obtain from naturally collected data (§2.3).

Stage III: multi-teacher on-policy distillation (MOPD). Domain-specialized models are consolidated into one policy. Merging independently trained experts directly causes interference, so teachers score trajectories the student itself sampled (§2.4). The underlying method is covered in On-Policy Distillation.

Putting the distillation signal where the action is

Vanilla on-policy distillation applies uniform token-level supervision across the whole response. But for a GUI agent, the only thing that touches the environment and changes state is the action. Most of the response is reasoning prose; the action is a short span. Allocate signal by token frequency and the part that matters gets under-supervised (§2.4).

Actions also have internal structure: the action type determines what the parameters mean. Coordinates are meaningful under Click; if the model wrongly predicted Scroll, its parameters are semantically irrelevant. The paper conditions the distillation signal on the correctness of the student's action while respecting that dependency (§2.4).

Stated in words, what the equation says is this: how the student's action went wrong decides where the correction lands and how hard it pushes. The signal is not spread evenly in proportion to token count; the weights are reassigned according to the kind of mistake.

Equation (1) rewrites the paper's three prose cases as a weight $w_t$, the strength of the distillation signal applied to the $t$-th token. Read it as three lines: if it was right, don't correct it; if the type was right but the coordinate was off, push hard on the action span; if the type itself was wrong, fix the type and ignore the parameters hanging off it.

There is a second trick on the teacher side. A hint $h(z^{*})$ giving the correct action type is appended to the teacher's prompt only. It never enters the student prompt and is unavailable at inference; the teacher does not generate its own response, it only uses the hint to modulate how it scores the student's tokens (§2.4).