Paper Walkthrough: Turning Game Development into a Verifiable Trajectory Data Engine — RLHEV and AWoMo

A paper arguing that what world models lack is not data or compute but cheap grading. We work through RLHEV — post-training on game-engine checks fused with developer accept/reject decisions — from first principles to equations, interactive figures, and the reported numbers.

Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models

Why "collect more video" will not scale world models

The standard recipe for making a world model stronger is to scrape more video, grow the model, and spend more compute. This paper flatly calls that recipe inefficient. What is missing, it argues, is not data or compute but a way to grade the output (§1).

Start with an analogy. A machine can mark a spelling quiz, because each answer is either right or wrong. It cannot mark an essay the same way. You can invent proxies for essay quality — word count, share of uncommon vocabulary — but if you optimise against them you end up with a long, jargon-stuffed, unreadable essay. The metric went up; the thing you cared about went down.

The paper says AI has split along exactly this line. Code is the quiz. Compilers and tests run, so correctness can be judged cheaply and at volume, which is what makes reinforcement learning post-training work. Spatial generation is the essay, graded only by fuzzy stand-ins like CLIP similarity, FVD, and MLLM-as-judge scores (§1, §3). The paper names the resulting cost the unverifiability tax: as long as you keep paying it, progress stays hostage to the sheer volume of data, compute, scans, and human annotation (§3).

The real reason code agents are winning

The paper does not explain code agents purely by "code is executable." It points to two verifiers stacked on top of each other (§1):

• Compilers, tests, and runtimes return dense, cheap feedback.
• The developer can reject a patch that passes every test, because it hurts maintainability or misses the product goal.

The first exposes local mistakes; the second decides global success. The paper calls this human-compiler dual verification, and argues it is the fuel that keeps improving a model long after pretraining ends.

Spatial generation has almost none of this. There is no cheap, reliable verifier that decides whether a generated video is physically and geometrically correct — whether objects persist through occlusion, whether the perspective is coherent (§3). 3D generation is short on raw data too: the paper puts the largest curated 3D asset corpus on the order of $10^{7}$ objects, against $10^{9}$ to $10^{12}$ images and tokens on the 2D vision-language side (§3). Learned world simulators face the hardest version, since supervising them needs real-world depth, geometry, and contact ground truth at a density that is very expensive to annotate (§3).

Why fuzzy grading caps how far you can scale

The paper makes the damage concrete with a plain decomposition. Let $Q^{*}$ be true quality and $R_{\mathrm{f}}$ the fuzzy reward you actually have (§2):

Here $x$ is the input, $y$ a candidate output such as a scene or 3D asset, $\varepsilon$ is zero-mean noise, and $b$ is systematic bias. Equation (1) just says: the gap between your proxy and the truth splits into random jitter plus a consistent lean in one direction. Put in words: the score you can actually compute misses the real quality by two separate amounts — a wobble that cancels out if you measure enough times, and a tilt that stays put no matter how often you measure.

The two harm you differently. Noise only lowers how much you learn per sample. Bias has a direction. If $b$ leans along an exploitable axis, maximising $R_{\mathrm{f}}$ raises proxy reward while lowering true quality (§2) — and piling on compute amplifies the exploit rather than fixing it. In the paper's framing, when reward is poorly grounded, the binding constraint on capability is not just data or compute but the fidelity and authority of the feedback signal.

The paper also proposes re-reading the Bitter Lesson along these lines (§2). Games, code, and mathematics did not advance only because general methods scale with compute. They advanced because each came with an evaluator that defines success — game rules, program execution, numerical checkers. On that reading, the cheapest remaining gains live in the feedback channel rather than in model architecture.

It helps to feel how arbitrary "similarity" grading really is.

A game engine is a compiler for space

Here is the proposal. The spatial verifier already exists — we just call it a game engine (end of §2).

A scene authored for Unity, Unreal, or Godot is not a picture; it is an executable specification, and the engine is its interpreter, runtime, and partial verifier. The checks the paper lists as cheap to run (§4):

• geometry and collision (collider penetration)
• physics and stability (does the rollout stay upright)
• navmesh reachability
• script errors and soft-locks
• bounded reachability or objective-completion probes under specified agents and seeds

The paper argues these are harder to game than appearance-based metrics for a blunt reason: a collision query is not fooled by a plausible texture (§4). The checks also form a ladder you can climb: does it load, is it physically plausible, is it functionally correct (navigable, objective achievable), and finally is it playable (§4).

The engine is not omniscient, though. It cannot decide whether a cutscene has the right mood (§2). So the final verdict stays with a human. That division of authority is the heart of the paper.

RLHEV: building reward from human and engine together

The method is RLHEV (Reinforcement Learning with Human-Engine Verification). Its reward abstraction is a constrained selection over candidates (§2):

$U_{H}$ is the human-review utility (typically 1 for accept, 0 for reject), $C_{i}$ are engine diagnostics such as penetration depth, $\phi_{i}$ is a penalty that vanishes when check $i$ is clean, $\lambda_{i}(h)$ weights that check in context $h$, and $G_{j}$ are hard gates you must pass or be disqualified. Equation (2) reads: among candidates that clear every mandatory engine gate, pick the one the human accepts with the fewest engine blemishes. It is a two-stage rule which says: let the machine throw out anything plainly broken (the gates $G_{j}$) before judging starts, then let a person decide accept or reject among the survivors, docking points for every rough edge the engine can still put a number on.