Paper Walkthrough: TurboVLA — Kick the LLM Out of the Loop and Run a Robot Policy at 32 Hz on an RTX 4090 with Under 1 GB of VRAM
Mainstream VLA models route every control step through a large language model. TurboVLA removes the LLM and fuses vision and instructions with lightweight bidirectional cross-attention instead — hitting 97.7% on LIBERO with 0.2B parameters, 31.2 ms latency, and 0.9 GB of VRAM. A from-scratch walkthrough of the paper.
TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM
Primary source — what this article is built on
undefined2026-07-29→undefined2026-08-13same month
TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAMHengyi Xie, Chenfei Yao, Xianjin Wu et al. · 2026-07-29 · v1arXiv:2607.27205Paper page·PDFundefined
Vision-language-action (VLA) models commonly adopt an LLM-centric $V \to L \to A$ pathway, where visual observations are projected into the representation space of a large language model before being decoded into robot actions. Although effective, this design incurs substantial computation and memory overhead at every policy invocation. In this work, we introduce TurboVLA, a new VLA paradigm that reformulates the conventional $V \to L \to A$ pathway as a direct $V + L \to A$ mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design constructs task-conditioned representations directly from visual and linguistic features, significantly reducing the computational and memory costs of VLA inference. On LIBERO, TurboVLA achieves 97.7% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GB inference VRAM on a consumer-grade RTX 4090, matching or outperforming substantially larger VLA policies. These results establish TurboVLA as a simple and effective alternative to the prevailing LLM-centric VLA paradigm, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.
The robot "reflex" problem
Pick up the red cup on the table. A human hand starts moving the instant the request lands. Today's AI-driven robots, by contrast, consult a giant language model for every step of that motion.
A model that takes camera images plus an instruction like "pick up the red cup" and outputs arm motions (a sequence of joint commands) is called a VLA — vision-language-action — model. In the mainstream design, exemplified by RT-2 and OpenVLA, images are first projected into the token space of a large language model (LLM), processed together with the instruction by the LLM, and only then decoded into actions. The paper calls this the V→L→A pathway (§1, §3).
The design has a real upside — it imports the broad semantic knowledge of LLM pretraining into robot control — but the price is steep: every single control step pushes data through billions of parameters. According to the paper's Table 1, OpenVLA needs 202.9 ms and 14.9 GB of VRAM per inference; even the streamlined π0.5 takes 93.6 ms and 12.8 GB (§5.3). Robot control is continuous, so slow inference means jerky motion, and big memory footprints mean the policy simply won't fit on real hardware without a large GPU.
As an analogy: a conventional VLA is like stopping to consult a professor, in writing, before every frame of movement — even though the instruction never changes during the task. Do we really need the professor every time?
Flipping the pathway: from V→L→A to V+L→A
TurboVLA starts from a simple observation: language is necessary for instruction-conditioned manipulation, but execution-level control need not be centered on a large language model (§1).
If the instruction "grab the red cup" already specifies which skill to perform, the policy doesn't need open-ended text generation or autonomous task decomposition. What it needs is to figure out how the current visual evidence should guide action, in light of the instruction. So: read the instruction with a lightweight text encoder like BERT, read the images with a vision encoder, and let the two streams exchange information directly through lightweight cross-attention. That is the paper's V+L→A pathway.
None of the ingredients is exotic — the bidirectional vision-language interaction is borrowed from grounding models like Grounding DINO and repurposed for action prediction (§1, §3). Skipping ahead to the punchline: this subtractive design reaches a 97.7% average success rate on the standard LIBERO benchmark with 0.2B parameters, 31.2 ms latency, and 0.9 GB of VRAM, matching or beating models more than fifteen times its size (§5.3). Over 30 policy invocations per second — hence the "32 Hz" in the title.
The heart of cross-attention is "dot product = similarity." Build that intuition first:
Mechanism, part 1: three lightweight encoders (§4.1)
TurboVLA splits its input into three streams, each with its own compact encoder.
The instruction goes through a lightweight text encoder (BERT by default), producing token-level features. The key decision: keep the full token sequence rather than pooling the sentence into one vector. Words like "red," "cup," and "left" carry object, attribute, and spatial-relation information that later stages need to match against specific image regions (§4.1).
The images are encoded per camera by a vision encoder (DINOv3 in the implementation, §5.1); each view's features get positional embeddings plus a camera-view embedding saying which camera they came from, and the streams are concatenated. Wrist-camera and third-person views stay distinguishable within a single sequence (§4.1).
Comments
Sign in to comment