Paper walkthrough: Qwen-Drive-1.0 — bolting 3D perception and planning onto a VLM without touching its architecture

Leave the pretrained VLM exactly as it is, attach a BEV perception head and a Planning Expert, and you get 3D detection, occupancy, maps and trajectory generation in one model — general vision-language ability nearly intact, 90.7 PDMS on NAVSIM. A ground-up walkthrough, straight from the paper.

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Driving well while staying smart

Autonomous driving has been moving away from task-specific modular pipelines toward unified learning-based systems. The current shape of that trend is the VLA (Vision-Language-Action) model: teach a large model that already understands images and language how to drive, and the broad knowledge from pretraining should carry it through rare situations.

The paper here comes from the Qwen Team and Huazhong University of Science and Technology. Its original title is "Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving" (arXiv:2609.00111, published 31 August 2026).

Here is what the abstract claims. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction and BEV map segmentation; it serves as a probe of the 3D information accessible from the shared representations, and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction following. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability, and evaluations across open-loop, pseudo-closed-loop and closed-loop settings show highly competitive motion-planning performance.

An analogy: teaching an interpreter to drive

Send a skilled simultaneous interpreter to driving school for three years and they will drive well — while their language skills quietly rust. Teach them from a textbook instead and they can say "move a little to the right" perfectly without knowing how many centimetres that is. This paper is best read as a collection of countermeasures against those two failures at once.

Words alone do not pin down 3D

The introduction (§1) names two limitations in the dominant recipe of adapting a general VLM through driving VQA.

First, textual targets do not directly constrain 3D layout, depth or occupancy. Even if the pretrained representation encodes spatial cues, language supervision neither requires an explicit 3D prediction nor lets you evaluate one. A model adapted this way can describe a scene fluently while remaining imprecise in 3D space.

Second, catastrophic forgetting. No finite driving dataset covers the rare situations met in deployment, which is exactly why pretrained knowledge matters — and exactly what aggressive domain adaptation erodes. The paper adds a deployment argument on top: production vehicles are consolidating cockpit and driving onto one compute platform, so a single model is expected to handle dialogue, instruction following and open-ended visual queries alongside driving. A model that trades general capability for driving scores forfeits that saving, because the cockpit then needs a second model and more compute.

Mechanism (1): the BEV head as a 3D probe

The backbone is Qwen3.5-4B, left untouched. Two external modules hang off it: the BEV perception head and the Planning Expert.

The BEV head takes the $N_v$ current surround-view images ($N_v \in \{6,8\}$ in the experiments) plus their calibrations and builds an ego-frame BEV representation (§2.1). It reads two feature streams: $\mathbf{F}^v_i$ from the vision encoder before the image tokens enter the VLM (low-level appearance), and $\mathbf{F}^m_i$ from those tokens after they traverse the full VLM (broader scene context). Lifting to 3D uses a depth-based view transform, in which a lightweight depth network predicts a categorical distribution $\mathbf{D}_i$ over $N_d$ depth bins per pixel — with no depth supervision at all.

In words, Eq. (1) smears each image feature along the camera ray, weighted by the predicted depth probabilities. $\Omega(\mathbf{p})$ is the set of views in which voxel $\mathbf{p}$ has a valid projection, $(u_i,v_i)$ its image coordinates, $d_i$ its depth bin. A peaked distribution concentrates the feature at one point on the ray; an unsure one paints it thinly along the whole ray.

The height-collapsed volume $\bar{\mathbf{V}}$ initialises the queries of a query-based BEV transformer, which aggregates a feature pyramid built from $\mathbf{F}^m$. The resulting BEV feature $\mathbf{B}$ feeds three branches: a DETR-style deformable decoder for detection, a shallow 3D UNet for occupancy (after expanding $\mathbf{B}$ along height and fusing it with $\mathbf{V}$), and a UNet-style head for map rasterisation. The loss is a plain sum, $\mathcal{L}_{\mathrm{perc}}=\mathcal{L}_{\mathrm{det}}+\mathcal{L}_{\mathrm{occ}}+\mathcal{L}_{\mathrm{map}}$. If BEV itself is unfamiliar, BEV representations from scratch makes this easier.

The probe result: 3D is not already in there

The most interesting experiment is not the top row of the results table but a couple of rows below it (§3.1, Tab. 1).

Freeze the VLM and vision encoder, train only the BEV head to convergence, and nuScenes performance stops at 35.60 mAP / 34.13 NDS — 6.34 mAP and 6.91 RayIoU behind BEVFormerV2*, a dedicated detector using the very same SigLIP-Qwen features. The paper's conclusion: vision-language-pretrained features support visual-text alignment but do not directly expose the 3D structure driving perception needs.