Paper Explained: OSReward — Can You Trust the AI That Grades AI? Remeasuring Rewards for Computer-Use Agents
The VLM judges that decide whether an agent driving a PC or a phone succeeded turn out to share a leniency bias that reads failure as success — a paper that exposes it with 1,019 human-verified trajectories and closes the gap with OS-Shepherd, an open reward model 30–60× cheaper, explained from zero background.
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Primary source — what this article is built on
undefined2026-07-30→undefined2026-08-13same month
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsQiushi Sun, Kanzhi Cheng, Yian Wang et al. · 2026-07-30 · v2arXiv:2607.28609Paper page·PDFundefined
Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60x lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.
The teacher who never looks at the answer sheet
Imagine a teacher who, when a student says "I got every question right," records full marks without ever looking at the paper. Most students are honest, but every so often one only believes they got it right. Grading that never checks the actual work turns that belief straight into a grade.
The paper we cover here — OSReward (University of Hong Kong, Nanjing University and others, arXiv:2607.28609) — points out that exactly this arrangement is running inside AI development today. The party being graded is the computer-using agent (CUA): an AI that operates a browser, a desktop, or a phone on a human's behalf to get tasks done. The party doing the grading is also an AI — a vision-language model (VLM) that can read both images and text. The paper is the first to systematically measure how trustworthy this "AI grading AI" setup is, and it shows that the graders share a habit of mistaking failure for success (§4.3).
Zero assumptions: what a trajectory and a reward signal are
A CUA receives an instruction like "open the expense screen and submit last month's report," then looks at the screen and repeats clicks and keystrokes. The record of that run — the screenshot at each step (the state), the agent's thinking, the action it executed — laid out in order is called a trajectory (§1).
Deciding whether a trajectory actually fulfilled the instruction is needed at every stage of CUA development. Performance evaluation, obviously; also filtering training data (keeping only the good trajectories); and reinforcement learning. In RL the verdict becomes the reward signal itself — the teacher for "reinforce this behavior or not" — so getting the verdict wrong means reinforcing the wrong behavior (§6.2). The judge is also called a reward model.
The problem is scale. Human-written verification programs only work on predetermined tasks and can't be applied to past trajectories whose environment no longer exists. Human annotation can't keep up with the volume. So in practice VLM-as-a-Judge — handing the trajectory to a VLM and asking for a success/failure ruling — became the de facto standard (§1). Yet the reliability of the judge itself went essentially unexamined. The paper's preliminary study finds that even the best VLM judge disagrees with the verifiers shipped with existing benchmarks on roughly a quarter of desktop verdicts (§1).
Building a benchmark for the judges, from scratch
To measure judges you need a set of trajectories with ground-truth verdicts attached. Recycling trajectories from existing benchmarks mixes in quality problems of the trajectories themselves plus verifier noise, leaving you unable to tell whether an error belongs to the judge or to the material (§3). So the authors built the entire collection pipeline themselves.
- Environments (§3.1): four platforms — Web (real sites), Windows, Ubuntu, Android. The desktops get roughly 20–30 applications, the way a real user's machine would, plus initialized user profiles, real files to edit, and deliberately confusable decoy content. The point is to create conditions where talking about completion isn't enough — you have to actually change the environment to succeed.
- Instructions (§3.2): annotators wrote them only after touching the environment for real, and cross-checks by someone other than the author removed the ambiguous and the impossible. About 800 survive out of roughly 1,500 candidates.
- Ground-truth labels (§3.3): three people judge each trajectory independently, and disagreements go to a meta-review by two senior reviewers. About 800 person-hours up to this point. The criteria are deliberately strict: an answer the agent never confirmed against the environment counts as a failure even when it happens to be correct.
The result is 1,019 human-verified trajectories (43% success / 57% failure, up to 100 steps) (§3.4). Two derived sets are carved out of it: OSReward-Hard (284 trajectories, 30% success / 70% failure), assembled by re-verifying the hard cases where the annotators themselves split, and OSReward-Multi, which layers fine-grained labels for faithfulness to intent (alignment) and absence of waste (efficiency) onto 440 successful trajectories (§3.4).
Comments
Sign in to comment