Paper Explained: Beyond Data Scaling — Why the Backbone, Not the Trajectory Count, Decides Your VLA (VLAct)

Robot trajectories can't be scraped off the web, so the paper argues you should squeeze reusable representations out of the trajectories you do have. Three modest interventions — preserving the VLM prior, co-supervising with multiple action heads, and partially unifying the action space across embodiments — beat industrial VLA systems on fully open data and 16 GPUs.

Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

Where "just collect more data" stops working

We tend to compress the success of language and vision models into one sentence: they read a lot. The paper pushes back on the shorthand. Web corpora worked not merely because they are large, but because they give broad coverage of visual and semantic variation (§1). Coverage, not volume, was doing the work.

Robots hit a wall right there. Robot trajectories cannot be scraped. They have to be produced by embodied execution in the physical world, usually under teleoperation or a carefully designed collection protocol (§1). Worse, the space a policy must generalize over is continuous and combinatorial — scenes times objects times task goals times embodiments times contact-rich dynamics. So even a very large robot dataset remains a sparse sample of the physical interaction space, with uneven coverage of the situations a robot will actually meet.

The paper therefore reframes the question. It is not an argument against scale. It is: given a fixed robot-data budget, how much transferable representation can you extract from those trajectories? That is an axis independent of how many you collect (§1). Put differently, treat continued pre-training not as large-scale action fitting, but as distilling trajectories into reusable visual-action knowledge inside the backbone.

What "continued pre-training" means here

Terminology first. Continued pre-training in this paper means: start from an already pretrained VLM and train it on broad, heterogeneous, multi-embodiment robot trajectories, before any downstream task-specific fine-tuning (§1). It is not pre-training a foundation model from scratch. The authors note that $\pi_0$, $\pi_{0.5}$, and GR00T N1/N1.5 all adopt exactly this setting and usually call it "VLA pre-training"; they simply use the more precise term.

Concretely: Qwen3-VL-4B as the base backbone, the StarVLA training codebase, and pre-training data drawn entirely from open-source robot datasets — DROID, InternA1, RoboCoin, MolmoAct — mixed with captioning data for representation preservation. Every experiment runs on 16 GPUs (§4).

Pilot study: the action head reshapes the backbone

In a VLA, the module that turns backbone features into actual actions is the action head. There are discrete token heads (FAST), regression heads (OFT), flow-matching heads (PI), and diffusion-style continuous heads (GR00T). Which one is best depends on the task, embodiment, horizon, and deployment constraints — there is no universally optimal head (§2).

So the authors froze the backbone choice at Qwen3-VL-4B and varied only the head used during pre-training and fine-tuning (§2). Two findings came out.

Discrete supervision transfers, but throws information away. Pair a FAST-pretrained backbone with a continuous GR00T head and you do slightly better than fine-tuning GR00T from scratch. But keeping the FAST head is much worse than continuous-head fine-tuning, and FAST pre-training does not close that gap. Discrete tokens teach coarse action structure while losing the fine-grained temporal and amplitude detail manipulation depends on.

A single continuous head collapses the backbone toward itself. OFT pre-training substantially helps when the downstream head is also OFT. Attach PI or GR00T to that same backbone and performance drops. The action information is probably still there — it has just been organized in a form only one decoder can read easily. The paper calls this decoder lock-in (§E.1) and draws the sharp conclusion: strong same-head performance overstates backbone reusability (§2).

VLAct's three prescriptions

From the pilot study the paper names three failure modes of naive continued pre-training: (1) robot trajectories are far narrower than web corpora, so end-to-end updating erodes broadly useful vision-language features; (2) supervision from a single head over-specializes the backbone to that head's decoding geometry; (3) embodiment-specific output spaces isolate actions that are physically comparable, like gripper open/close (§1). VLAct answers each.

1. Protect the VLM prior

Freeze the entire vision encoder and the lower half of the LLM layers during pre-training; update only the upper layers and the action heads. The split protects low-level visual processing and early vision-language alignment while letting the upper layers adapt to action-conditioned reasoning. Downstream fine-tuning unfreezes everything. This freezing alone is worth 3.7% on LIBERO-Plus and 3.4% on Agilex (§3.2).

The second mechanism is caption mixing. Blending VLM data into VLA pre-training is standard, but the paper compares mixtures and reports that image captions are the strongest anchor — captions give dense supervision over objects, attributes, spatial relations, and scene context, which is exactly the prior being protected (§3.2).