2026-08-13·FREE·PAPER·10 min read

Paper Explained: OSReward — Can You Trust the AI That Grades AI? Remeasuring Rewards for Computer-Use Agents

The VLM judges that decide whether an agent driving a PC or a phone succeeded turn out to share a leniency bias that reads failure as success — a paper that exposes it with 1,019 human-verified trajectories and closes the gap with OS-Shepherd, an open reward model 30–60× cheaper, explained from zero background.

#agents #computer-use #reward-model #vlm #llm-as-judge #paper

ModalityimageTaskagents

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Primary source — what this article is built on

undefined2026-07-30→undefined2026-08-13same month

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsQiushi Sun, Kanzhi Cheng, Yian Wang et al. · 2026-07-30 · v2arXiv:2607.28609Paper page·PDF

undefined

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60x lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.

The teacher who never looks at the answer sheet

Imagine a teacher who, when a student says "I got every question right," records full marks without ever looking at the paper. Most students are honest, but every so often one only believes they got it right. Grading that never checks the actual work turns that belief straight into a grade.

The paper we cover here — OSReward (University of Hong Kong, Nanjing University and others, arXiv:2607.28609) — points out that exactly this arrangement is running inside AI development today. The party being graded is the computer-using agent (CUA): an AI that operates a browser, a desktop, or a phone on a human's behalf to get tasks done. The party doing the grading is also an AI — a vision-language model (VLM) that can read both images and text. The paper is the first to systematically measure how trustworthy this "AI grading AI" setup is, and it shows that the graders share a habit of mistaking failure for success (§4.3).

Zero assumptions: what a trajectory and a reward signal are

A CUA receives an instruction like "open the expense screen and submit last month's report," then looks at the screen and repeats clicks and keystrokes. The record of that run — the screenshot at each step (the state), the agent's thinking, the action it executed — laid out in order is called a trajectory (§1).

Deciding whether a trajectory actually fulfilled the instruction is needed at every stage of CUA development. Performance evaluation, obviously; also filtering training data (keeping only the good trajectories); and reinforcement learning. In RL the verdict becomes the reward signal itself — the teacher for "reinforce this behavior or not" — so getting the verdict wrong means reinforcing the wrong behavior (§6.2). The judge is also called a reward model.

The problem is scale. Human-written verification programs only work on predetermined tasks and can't be applied to past trajectories whose environment no longer exists. Human annotation can't keep up with the volume. So in practice VLM-as-a-Judge — handing the trajectory to a VLM and asking for a success/failure ruling — became the de facto standard (§1). Yet the reliability of the judge itself went essentially unexamined. The paper's preliminary study finds that even the best VLM judge disagrees with the verifiers shipped with existing benchmarks on roughly a quarter of desktop verdicts (§1).

Building a benchmark for the judges, from scratch

To measure judges you need a set of trajectories with ground-truth verdicts attached. Recycling trajectories from existing benchmarks mixes in quality problems of the trajectories themselves plus verifier noise, leaving you unable to tell whether an error belongs to the judge or to the material (§3). So the authors built the entire collection pipeline themselves.

Environments (§3.1): four platforms — Web (real sites), Windows, Ubuntu, Android. The desktops get roughly 20–30 applications, the way a real user's machine would, plus initialized user profiles, real files to edit, and deliberately confusable decoy content. The point is to create conditions where talking about completion isn't enough — you have to actually change the environment to succeed.
Instructions (§3.2): annotators wrote them only after touching the environment for real, and cross-checks by someone other than the author removed the ambiguous and the impossible. About 800 survive out of roughly 1,500 candidates.
Ground-truth labels (§3.3): three people judge each trajectory independently, and disagreements go to a meta-review by two senior reviewers. About 800 person-hours up to this point. The criteria are deliberately strict: an answer the agent never confirmed against the environment counts as a failure even when it happens to be correct.

The result is 1,019 human-verified trajectories (43% success / 57% failure, up to 100 steps) (§3.4). Two derived sets are carved out of it: OSReward-Hard (284 trajectories, 30% success / 70% failure), assembled by re-verifying the hard cases where the annotators themselves split, and OSReward-Multi, which layers fine-grained labels for faithfulness to intent (alignment) and absence of waste (efficiency) onto 440 successful trajectories (§3.4).

FIG 1Read the x-axis as the number of steps in a trajectory. The more steps, the more places a failure could be hiding, and the last few screenshots stop being enough to check. In the paper too, failed trajectories are distinctly longer than successful ones (§3.4)

Finding 1: A ceiling on accuracy — only the frontier clears the bar

Twenty-seven VLMs are put on the bench under identical conditions. The protocol is shared: the judge reads the last five screenshots of the trajectory plus the thinking and action text of every step, and outputs success or failure (§4.1). Beyond plain accuracy, the metrics are $\mathrm{sRec}$ , the share of successful trajectories correctly called successes (low means too strict), $\mathrm{fRec}$ , the share of failed trajectories correctly called failures (low means too lenient), and their mean.

\mathrm{BalAcc} = \frac{\mathrm{sRec} + \mathrm{fRec}}{2}

(1)

In words: $\mathrm{BalAcc}$ — balanced accuracy — is the two rates added together and halved, nothing more. $\mathrm{sRec}$ is how often the judge calls a success a success, $\mathrm{fRec}$ is how often it calls a failure a failure, and giving each exactly half the weight is what makes the number honest: even with the success/failure mix skewed 43 to 57, you can't run up a score by getting only one side right (§4.1).

The result is that only frontier models come close to the practical bar for a reward model (the paper works with roughly 90%). Claude-Opus-4-8 leads at 89.7%, GPT-5.5 and Claude-Opus-4-6 follow at 89.5%, and there's about a 20-point gap down to the small open VLMs (§4.2, Table 1).

Finding 2: The judges are lenient across the board — the dominant error is missing false successes

What matters more is the direction of the errors. Plot each judge on the (fRec, sRec) plane and a large "lenient cluster" appears — a group that accepts almost anything as a success and can't catch failures (§4.3). Classify the wrong answers and "over-accepting incomplete tasks" alone accounts for two-thirds of all errors. And this is not one model's quirk: it is the single largest error mode in all 27 judges (at least 48% of every judge's own errors). Over-acceptance outnumbers over-rejection 3 to 1 overall (§4.3).

The typical case is a false success — a trajectory where the agent announces "task completed" while having actually failed. The hard set OSReward-Hard concentrates trajectories of this kind, and the judges fall apart on it. Even the best, Claude-Opus-4-8, drops 20 points to 69.7% — the accuracy a machine that simply always answers "failure" would get at a 30/70 split. The average judge lands at 52%, barely above a coin flip (§4.4). By platform, Windows is the hardest and mobile the easiest; by failure type, missed perception and manipulation errors — the ones you can only catch by reading the screen — are markedly harder than planning errors that surface in the text (§4.4).

Why the leniency: the judges were reading self-reports, not screens

The paper ablates the input one element at a time. Cutting the screenshots from five to three, or removing the red markers on click positions, doesn't move accuracy by even half a point. Sweeping the count from 1 to 16 shows no trend at all (§5.1). But dropping the per-step thinking and action text costs 7.2 points on average and flips 22.7% of the verdicts — three times the impact of any change on the visual side (§5.2).

In other words, the judges put their weight on the agent's text (its self-report) rather than on the screen (the evidence). That is structurally an invitation to be taken in by a confident "task completed", and the paper concludes this is the mechanism behind the leniency bias (§5.2). Back to the opening analogy: they were grading the student's claim, not the answer sheet.

Can you fix it by changing how you run them? Turning up thinking mode helps less the stronger the judge (+2.8 pt for the weakest, +0.4 pt for the strongest). Majority voting across judges doesn't help either — the judges make the same mistakes on the same hard cases (agreement among the top judges is κ≈0.71), so a majority vote of the top three beats the best single model by about one point at several times the cost (§5.3). Meanwhile the ideal ceiling of "count it if any one of them got it right" is 99%. The correct answer is in the pool; majority voting simply can't pick whom to trust (§5.3).

FIG 2Raising the temperature flattens the output distribution, so options with close probabilities swap places more easily. In the paper, merely re-judging the same trajectory at temperature 0.7 flips 6–9% of the verdicts (§5.3). Aggregate accuracy stays stable, but the individual labels — consumed one at a time as rewards — take that jitter head-on

The cost wall, and OS-Shepherd as the answer

A trustworthy judge is expensive. Judging all 1,019 trajectories costs about $100 with Claude-Opus-4-8 and $45 with GPT-5.5, while judges under $3 sink to 57% or below on Hard (§5.4). For reinforcement learning that's fatal: by the paper's estimate, one modest RL run (200 updates × batch 16 × 16 rollouts) needs 51,200 verdicts on its own — about $4,000 with Opus (§6.3).

So the authors build their own judge. Training data first. Human labels are impossible at that scale, so they auto-label with an agreement filter that keeps only the trajectories on which diverse strong judges independently reached the same verdict (§6.1). If majority voting doesn't work, throw the split cases out entirely — unlike a benchmark, training data is free to discard. That's the reversal of perspective. From trajectories produced by agents across five model families they build and release OS-Shepherd-100K, a corpus of about 100,000 items with written rationales for each verdict (§6.1).

Training runs in two stages (§6.2). First SFT (supervised fine-tuning) on 96.6K items to imitate the verdicts, which corrects the base model's nearly across-the-board leniency. Then they aim at the remaining sore spot — false successes — mining roughly 3.1K cases where the SFT model "gets it wrong under greedy decoding but can get it right when sampling," and reinforcement-learning on those with GRPO. The resulting OS-Shepherd-9B reaches 86.1% overall and 60.2% on Hard, a large gain over plain Qwen3.5-9B (76.7% / 39.4%). Failure detection on Hard in particular goes from 14.1% to 57.6% (Table 4). Judging the whole set costs $1.36 — 30–60× cheaper than the frontier, and better than commercial judges in the same price band (§6.3). Tellingly, a 35B version gains only +2.4 pt of BalAcc on Hard: what's doing the work is the recipe, not the size (§6.3).

Generalization is checked too. On OSWorld, AndroidWorld, and WebArena — three benchmarks never used in training — OS-Shepherd is the best open judge on the first two and beats Qwen3.5-397B, roughly 44× its size, on all three. What transfers is not knowledge but the debiasing itself: the ability to catch failures (§7.2).

Limitations

The paper is equally clear about what shouldn't be oversold. First, OS-Shepherd doesn't reach frontier accuracy (60.2% on Hard vs. Opus's 69.7%). Second, fine-grained scoring of the quality of a success (alignment/efficiency) is a weakness of the judges as a whole, and OS-Shepherd-9B stays near the constant baseline (§4.5, §6.3). Third, the training corpus's labels come from an ensemble of judges rather than from humans, so error types on which the judges agree can in principle contaminate it (§6.1). Comparisons against external benchmarks are also "agreement rates with that verifier," which carry the verifier's own imperfections (§7.1). The arXiv comment likewise says "Work in progress," leaving additional model sizes and the like to future work (Appendix A).

How this is used in practice

For anyone using an LLM/VLM judge as a reward or a quality gate in agent development, this paper doubles as a design checklist.

If you're choosing a reward model for RL or trajectory filtering: look at fRec (failure detection rate), not overall accuracy. The paper's central lesson is that judges exist whose aggregate accuracy looks fine but which are fatal as rewards. Build a few dozen false successes from your own data — trajectories that claim completion while having failed — and measure detection rate on them before you adopt anything.
If you're writing judge prompts: what you may cut from the input is markers and surplus screenshots; what you must not cut is the per-step action text (§5.2 recommends keeping the full text history, dropping the markers, and tuning the screenshot count between 5 and 9 per model). And always be suspicious of the structure in which a judge over-trusts the agent's self-reported text.
If you auto-label training data: use a high-agreement filter and discard the splits, not majority voting. Forcing labels onto trajectories where judges disagree bakes the judges' shared bias into your data (§6.1).
The pitfall that becomes an incident: take a verdict just once with temperature sampling and each individual label fails to reproduce 6–9% of the time (§5.3). For aggregate evaluation that's ordinary noise; in a pipeline that consumes labels one at a time as rewards it does real damage. Pin greedy decoding, or confirm agreement across multiple samples.

LLM Agents from Scratch — the basic agent loop, and why verification is needed
Instruction Tuning and RLHF — where the idea of a reward model starts
Paper Explained: LongHorizon-Harness — a paper that implements "don't believe the self-report" on the harness side