Paper explained: DEFT-RLVR — show a driving VLM the future trajectory too early and it fabricates its reasoning

Let a teacher model peek at the ground-truth trajectory while writing chain-of-thought annotations for driving, and severe hallucinations double from 29% to 50% — the paper calls this trajectory anchoring bias. This article walks through AD-MCQ, which turns planning into a verifiable multiple-choice task, and DEFT-RLVR (arXiv:2608.01755), which reveals trajectories only after the model commits to a decision, using nothing but the paper itself as the source.

Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs

A teacher who sees the answer first invents the explanation

Imagine asking a driving instructor: "The correct move here is a right turn. Now explain why." Instead of retracing an actual decision process, the instructor will construct a justification for the turn — and in the worst case, invent a "mandatory right-turn sign" that was never there.

That last part is not hypothetical. Figure 1 of this paper (arXiv:2608.01755) shows a real example: a teacher model that was shown the correct trajectory fabricated a mandatory-turn sign absent from the scene to make its explanation work (§1).

Mainstream VLA (Vision-Language-Action) models for autonomous driving pair a large VLM (vision-language model), which handles high-level reasoning and decision making, with a much smaller action expert specialized for geometric trajectory prediction (§1). Think of it as the classic end-to-end driving stack (End-to-End Driving from Scratch), which pushes pixels to control through one network, with a language layer bolted on so the decisions can be explained and checked. To sharpen the VLM's reasoning, the standard recipe is to have a teacher model write chain-of-thought (CoT) rationales as training data. The problem: most existing annotation pipelines show the teacher the ground-truth (GT) future trajectory before it writes the rationale. The paper argues, and then demonstrates, that this induces trajectory anchoring bias — the cognitive-psychology phenomenon where information supplied up front disproportionately shapes the judgment that follows (§1, §2).

The evidence: severe hallucinations double from 29% to 50%

The paper's controlled study is meticulous (§2, §B.1). The authors selected 100 causally demanding scenes — hard braking, stopping from motion, sharp turns — and used the same teacher (Qwen3.5-397B-A17B), the same 12 video frames, a byte-identical system prompt, and temperature 0. The only difference between the two arms is one contiguous block: the 10 raw waypoints of the logged future trajectory, present in one arm and absent in the other.

Two human annotators, blind to which condition produced each output, scored all 200 CoTs on grounding, absence of hallucination, specificity, and causal coherence (§B.2):

• Severe hallucination rate: 29.0% without the trajectory, 50.0% with it (Table 1)
• Pairwise preference: rationales written without the trajectory win 60.5% of judgments; with it, only 24.0% (Table 1)

Handing the model the answer up front doesn't improve its explanations — it pushes the model toward fabricating scene evidence to justify a foregone conclusion. And the degradation concentrates exactly where reliable reasoning matters most: the causally hard scenes (§1).

The obvious fix — hide the trajectory and let the model decide from the scene — creates a new problem. Open-ended trajectory generation entangles high-level decision making with millimeter-scale geometric synthesis and low-level vehicle dynamics in a single task (§1), which is a poor match for what language models are good at.

Tool number one: turn driving decisions into a multiple-choice question — AD-MCQ

So the paper converts planning into a multiple-choice question. AD-MCQ (Autonomous-Driving Multiple-Choice Question) presents each scene with a small set of explicit candidate trajectories and asks the model to pick one (§3). A choice can be verified exactly, by machine.

Where do candidates come from? The authors take roughly 490,000 logged 5-second trajectories (10 waypoints at 0.5-second intervals), run K-means, and obtain a codebook of $K{=}8192$ motion prototypes. Any trajectory is discretized by assigning it to its nearest prototype (§3):

Put in words: walk through all ten time steps, measure how far your trajectory's waypoint $\mathbf{p}_t$ sits from prototype $k$'s waypoint $\mathbf{c}_{k,t}$, add those gaps up, and keep the prototype number $z$ whose total is smallest. It is a nearest-neighbour lookup — "which of the 8,192 stock motion shapes is closest to the one actually driven?" The choice of $K{=}8192$ balances reconstruction fidelity (0.290 m average error even on unseen trajectories) against prototype utilization (92.0%) (§3, §C.5).

Each question mixes the one correct prototype with hard negatives — wrong answers whose similarity $\rho$ to the oracle falls in a controlled band — into a six-option set. Because the options preserve trajectory-level distinctions in braking time, speed profile, and lateral geometry, the task probes far finer judgment than coarse meta-actions like "turn left / go straight" (§3).