Paper Walkthrough: GigaBrain-0.7 — 37,000 Hours of Embodied Experience and a Three-System Robot Brain

A ground-up walkthrough of GigaBrain-0.7, which splits a robot foundation model into understanding, prediction/evaluation, and action, then pretrains it on 37,256.98 hours of embodied data in a single stage. Covers subgoal-image and progress-value conditioning, Soft Knowledge Insulation, and the real-robot numbers — strictly from the paper.

GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture

The robot that cannot fold a shirt

"Fold that shirt." A person starts within seconds; a robot finds this brutally hard, for three reasons. The shirt changes shape every time you grab it, so replaying a memorized trajectory gets you nowhere. The scene keeps returning to visually similar states, so a single frame cannot tell you which pass you are on. And when things go wrong, the robot has to notice that on its own and recover.

The dominant approach today is the VLA (Vision-Language-Action) model: take a large model that understands images and language, and have it emit actions (see our TurboVLA walkthrough). The paper's complaint starts one step past that. Current VLAs stay centered on reactive observation-to-action prediction, with little machinery for anticipating what happens next, and little for judging whether the current behavior is actually working (§1). GigaBrain-0.7 tries to fill that gap with three systems that each do a different job.

The metaphor: three cooks in one kitchen

Picture a busy kitchen run by three people.

• The planner (System 2) looks at what is on the counter and breaks the order down into "pick up the red hat."
• The taster (System 3) predicts "in a few seconds it should look like this," and scores whether the current attempt is getting closer to done.
• The hands (System 1) take that instruction plus the prediction and the score, and actually move the joints.

In the paper's terms these are understanding and planning, prediction and evaluation, and action and control. Concretely: System 2 is a PaliGemma2 (3B) VLM, System 1 is the same PaliGemma2 (3B) plus a dedicated Action Expert (0.5B), and System 3 is a 5B world value model built on GigaWorld-1 (§4).

What matters is that the three are trained under separate objectives and exchange information only through fixed, narrow interfaces. System 2 hands over a sentence describing what to do next; System 3 hands over one predicted future image and a single bit for "is progress going up." Nothing vague flows between them, which means you can tell which part is helping and which part broke.

"Bigger backbone, better robot" does not hold

So which VLM belongs underneath System 1? The paper compares PaliGemma2 (3.5B total), Qwen3.5 (5B), and Gemma 4 (8.5B) on real-robot tasks (§6.2, Tab. 4). The answer is not tidy. Gemma 4 scores best on desk cleaning and fruit picking — and 0% on shirt folding. PaliGemma2 is the only backbone with a nonzero shirt-folding rate (30%), which is why the paper builds on it.

The paper flags its own caveat: the configurations differ in both model size and image resolution, so this is not a controlled scaling study (note to Tab. 4). Still, you can read off where the authors chose to invest — not in a bigger backbone, but in data and in division of labor.

Data: forcing 37,256.98 hours into one shape

After cleaning, the embodied trajectory corpus totals 37,256.98 hours: real robots 20,535.65 h (55.12%), UMI hand-held human demonstrations 8,251.83 h (22.15%), EGO first-person human video 2,862.36 h (7.68%), simulation 1,453.92 h (3.90%), and world-model-generated data 4,153.22 h (11.15%). The real-robot slice alone spans 16 robot types, 1,810,101 episodes, and 2,077,837,071 frames, and runs alongside 271,976,674 image-text/VQA samples (§3.1).

Heterogeneous data interferes if you just pile it together, so the normalization steps are spelled out: convert everything to LeRobot v3.0, fix the dimension ordering as left arm → right arm → head → waist → base or legs, represent rotations in the continuous 6D format, and rewrite language instructions with GLM-5.1 (§3.2). Cleaning is four-stage: q01/q99 outlier removal (quantile normalization is used in training, so extreme values stretch the range and squash normal action signals), removal of long stationary segments, end-effector pose alignment using collected URDFs, and tracing abnormally high training losses back to the source video to blacklist the sample.

Inside System 1, and the trick of throttling the gradient

System 1 is a Mixture-of-Transformers that runs a vision-language (VL) stream and an Action Expert (AE) stream side by side at every layer (§4.2). The VL stream keeps causal self-attention so the pretrained VLM's autoregressive interface stays intact; the AE stream attends bidirectionally over the concatenation of both streams.

$\mathbf{H}^{\mathrm{VL}}_{l}$ is the vision-language hidden state at layer $l$, $\mathbf{H}^{\mathrm{AE}}_{l}$ is the action-side hidden state, and $[\cdot,\cdot]$ means concatenating them. Stated in words: the action side can read the language side's notes at every layer, but not the other way around — the attention query comes from the action stream alone, while the keys and values come from both streams stacked together. Actions themselves are produced by flow matching, and that loss is optimized in the same pretraining pass as next-token prediction for language, subtasks, and discrete actions (§5.1). Not splitting these into phases is what the paper calls one-stage alignment training.

The hard part is teaching physical control without eroding the language ability you started with. Let the action gradient through unchecked and language understanding thins out; block it entirely and the VLM's visual features never adapt to embodied control. Soft KI sits in between: the flow-matching gradient flows at full strength inside the Action Expert, but is scaled by $\alpha_{\mathrm{KI}}$ just before it reaches the VLM backbone (§5.1).

$\mathcal{L}_{\mathrm{NTP}}$ is the next-token-prediction loss, $\mathcal{L}_{\mathrm{FM}}$ the continuous-action loss, and $\theta_{\mathrm{VL}}$ the backbone parameters. Read it as: the language learning signal arrives at full volume, the action signal comes through a partly closed tap. $\alpha_{\mathrm{KI}}=0$ is full insulation, $1$ is no insulation.

Read in words, the equation says that the direction the backbone moves in is the language learning signal at full size plus a shrunken copy of the action learning signal. The left-hand side is the update the VLM actually takes; the two terms on the right are where that update comes from; and $\alpha_{\mathrm{KI}}$ is the dial for how loudly the action side gets to speak.

System 1 also carries short-term visual memory. Temporal-Spatial Blocks in the visual encoder aggregate recent frames along time, then process space, and discard the past-frame tokens, keeping only the enriched current frame (§4.2). Adding history therefore does not grow the token count entering the backbone.

System 3: one image of the future, one bit of progress

System 3 exposes exactly two signals (§4.1). The first is a subgoal image $g_t$: conditioned on the current observation, proprioceptive state, and the subtask sentence, the world model generates a short future video, and the last frame is taken as the subgoal. The second is an evaluation of progress — a scalar $V_t$ estimating how far execution has advanced toward subtask completion, collapsed to a bit by the sign of its difference (§5.3).

$\mathbb{1}[\cdot]$ returns 1 when the bracketed condition holds and 0 otherwise, and $\delta_t$ is the gap between two time points inside the action chunk. All it extracts is: did the work move forward over this segment, or stall and slip back? Put in words: $V$ is a score for how close the subtask is to done, and $A_t$ is a switch that returns 1 if that score is higher a little later and 0 if it is not. The size of the score never reaches the policy — only the single bit for up or down. During training the true value is used; at inference the policy is always conditioned on the "moving forward" setting.

System 3 is frozen after its own pretraining — during VLA post-training it produces fresh $g_t$ and $V_t$ per sample but receives no gradient (§5.2). The two conditions are also dropped independently with probability 0.50 and 0.15 (§5.3), so the policy experiences all four regimes (both, image only, value only, neither) and still runs when System 3 is unavailable (see also our PhiZero walkthrough).

# Skeleton of one post-training step, following the description in §5.3
g    = world_model.subgoal_image(obs, state, subtask)  # last frame of predicted video
A    = 1 if world_model.value(t + dt) - world_model.value(t) > 0 else 0  # eq. (3)
if rand() < 0.50: g = None     # drop the subgoal image
if rand() < 0.15: A = None     # drop the progress condition
ctx  = (obs_hist, lang, proprio, robot_id, g, A)
loss = flow_matching_loss(action_expert(ctx, a_tau, tau), a, eps)  # System 3 stays frozen

What the experiments show

Data scale (§6.3). More pretraining data consistently lowers validation loss. But real-robot gains depend on the task: fruit manipulation improves smoothly, while chaotic clothes folding depends far more sharply on data scale. Starting from 3,000 hours of robot data, adding equal amounts of UMI or EGO improves things individually, and all three together is strongest. Because that gap survives post-training, the paper argues human data changes the quality of the pretraining prior rather than giving a one-off boost.

Coupling strength (Tab. 5). The layer-interleaved Dual Stream is best on all three tasks (desk 50% / fruit 88% / shirt 30%) but costs 0.221 s per inference step. Last-layer cross attention runs at 0.073 s and scores 20% / 40% / 0%. Speed and capability trade off head-on.

What System 3 buys (§6.5.2). On clothes folding all four configurations saturate at 100% success — yet average task score climbs from 68.3% (Base) to 81.7% / 85.0% / 88.3% (+SubImage / +Value / both), and completion time falls from 107 s to 83 / 79 / 75 s. Gift wrapping separates them further: the base model never completes the task (0%), versus 20% / 60% / 80%. The claim is that these signals improve the quality of intermediate progress, not just the binary outcome.

Baselines and RL (Tab. 6, 7, 9, 12). After post-training, the six-task language-following average is 91.5% on AgileX PiPER ($\pi_{0.5}$: 88.8%) and 84.2% on Maker H01 (75.2%). Complex manipulation averages 84.9% on PiPER (76.6%) and 74.1% on H01 (45.2%). On RoboTwin 2.0 the model reaches 67.9% under domain randomization (Hard) versus 46.0% — losing in the clean setting (66.8% vs 70.7%) but flipping the result on Hard. And running four real-robot tasks through SFT → offline RL → online RL moves average success from 30.0% to 57.5% to 100%.

What the paper does not claim

• OOD degradation is stated openly. Both platforms drop under out-of-distribution settings, and the difficulty compounds when object identity, scene, and task composition shift at once (§6.6.1). This is shown through representative rollouts, not a quantitative table.
• The backbone comparison is not a controlled experiment. Size and resolution vary together, and Gemma 4's desk cleaning was evaluated under a limited-range setting (note to Tab. 4).
• The final MiMo-Embodied score of 0.5704 carries a caveat. It comes after MiMo data entered the training mixture, and the paper itself treats it as a continued-pretraining diagnostic rather than a held-out result; 0.4621 is the pre-contamination number used for comparison (§6.8.1).
• Trial counts are small. Evaluation uses roughly 10–20 trials per configuration (§6.1), which is the context for "100% on every task." The human-intervention protocol and the full online-RL formulation are explicitly deferred to a future report (§5.4), and the paper notes contact-rich tasks remain far from saturated (rice sweeping on PiPER sits at 40%).

How this plays out in practice

If you are post-training it. The knobs are the action-chunk horizon, the condition dropout rates (0.50 for the subgoal image, 0.15 for the value signal in the paper), and whether inference pins the progress condition to "advancing." System 3 is frozen, so only Systems 1 and 2 are optimized — get that wrong and you push useless gradients into the world model. The other real constraint is control frequency: Dual Stream's 0.221 s per step will not drop into a high-rate control loop unmodified. Last-layer cross attention buys you 0.073 s, with 0% shirt folding attached to it.

If you are building the data pipeline. Most accidents happen in normalization and coordinate frames. A single surviving q01/q99 outlier stretches the normalization range and squashes normal actions into a narrow band, destabilizing optimization. If end-effector reference points are defined differently per platform, the same "grasp" turns into different numbers. And without a fixed dimension ordering plus validity masks, an absent dimension becomes indistinguishable from a genuine zero.

If you are reading the numbers. Look only at success rate and System 3's contribution to clothes folding vanishes entirely (100% everywhere). It shows up only once you also report average score and completion time — that is the transferable lesson here. Conversely, do not treat 40% vs 55% over 10–20 trials as a real difference. Read benchmarks with Easy and Hard separated (this paper loses on Easy and wins on Hard), and cite leaderboards with their snapshot date.