2026-09-11·★ MEMBER·PAPER·8 min read

GE-Act 2.0 Paper Explained: Pretraining and Scaling a World-Action Model for Robotic Manipulation

GE-Act 2.0 is a world-action model trained from scratch on manipulation data instead of inheriting a video generator. We walk through its single-step visual planner, the KASO rule that keeps only futures compatible with the recorded action, and what 100 zero-shot tasks actually showed.

Key points

A world-action model predicts how the scene will change next, then uses that prediction to produce joint commands.
GE-Act 2.0 initializes every trainable generative and action component from scratch on manipulation data, combining CoAE, SVP and IDM and connecting them with KASO.
With no per-task fine-tuning, scaling co-training data from 300 to 30,000 hours raised success from 17.1% to 44.1% on G1-OP.
CoAE downsamples each spatial axis by $64\times$ , mapping a $256\times384$ frame to a $4\times6$ grid of 24 tokens per frame.
CoAE reaches an action MAE of 0.01673, trailing DINOv3 at 0.01273 with one-sixteenth the tokens, and beating DC-AE at 0.02573.

#world-model #robotics #manipulation #flow-matching #pretraining

ModalityimageTaskgeneration

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

Primary source — what this article is built on

undefined2026-09-04→undefined2026-09-11same month

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic ManipulationAgiBot Research Team, Renhang Liu, Wenzhi Zhao et al. · 2026-09-04 · v1arXiv:2609.05588Paper page·PDF

undefined

World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.

Rehearsing the motion before you move

Before reaching for a cup, you run a short film in your head: approach like this, close the fingers there. World-action models (WAMs) ask a robot to do the same thing — predict how the scene will change next, then use that prediction to produce joint commands.

The paper is titled "GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation" (arXiv:2609.05588, AgiBot Research Team). Its abstract, in short: because a world-action model predicts future states in order to guide actions, it can learn from action-free video as well as from action-labeled interaction. Yet most existing systems inherit a pretrained video generator, leaving the pretraining and scaling of the world-action model itself unexplored. GE-Act 2.0 initializes every trainable generative and action component from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM), then connects them with KASO (knowledge-aligned selective optimization). Evaluating pretrained checkpoints directly, with no per-task fine-tuning, scaling co-training data from 300 to 30,000 hours raised success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D.

The paper frames three design problems (§1). Representation — how to build a latent space that keeps future prediction cheap while retaining the information an action model needs. Generation — how to exploit two very different data types: action-free video, which visual pretraining can absorb at a scale robot demonstrations never reach, and instruction-free robot trajectories such as failed attempts and deployment rollouts, which conventional policy objectives struggle to use. Alignment — how to make a generated future agree with a recorded action. The sections below follow that order.

Part 1: a compressor built for control, not for reconstruction

The autoencoders used by video generators are trained to reconstruct nicely. What a world-action model actually needs is for the compressed state to still reveal what the robot did — and, because every future prediction happens inside the control loop, for the token count to stay small. CoAE rebuilds this piece. Where common video autoencoders downsample each spatial axis by $8\times$ or $16\times$ , CoAE uses $64\times$ , mapping a $256\times384$ frame to a $4\times6$ grid — 24 tokens per frame (§3.2.1). Training combines pixel, perceptual (LPIPS) and adversarial reconstruction losses with alignment to three frozen visual teachers: the SigLIP 2 encoder used by Qwen3.5, V-JEPA 2.1, and DINOv3.

\mathcal{L}_{\mathrm{align}}=\sum_{k=1}^{3}\lambda_{k}\bigl(1-\cos(g_{k}(z),G_{k}(o))\bigr)

(1)

In words, this pulls the compressed code toward the features each of the three frozen teachers extracts from the same image, so that the two point in the same direction. Cosine measures how well the directions agree, and each teacher gets its own weight.

In plain terms: push the compressed code $z$ , after a small learned head $g_k$ , to point in the same direction as the features a teacher $G_k$ extracts from the same image. $\cos$ measures directional agreement and $\lambda_k$ weights each teacher. The three teachers contribute language-aligned semantics, spatiotemporal structure, and dense visual features respectively.

FIG 1Compress harder and the picture falls apart. But "can you still read the action off it" is a different axis from visual prettiness — that is CoAE's whole argument

A controlled probe backs this up (§3.2.2). Freezing each representation and training the same inverse-dynamics probe, CoAE reaches an action MAE of 0.01673. It trails the best representation, DINOv3 at 0.01273, but uses one-sixteenth as many tokens per frame. DC-AE, which uses the same $64\times$ compression, sits at 0.02573 — 54% worse.

GE-Act 2.0 Paper Explained: Pretraining and Scaling a World-Action Model for Robotic Manipulation

Key points

Rehearsing the motion before you move

Part 1: a compressor built for control, not for reconstruction

Members-only from here

References

Published since

Comments