Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?

What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget.

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Teaching three subjects to one student

Imagine teaching one student three subjects at once: language, art appreciation (understanding images), and painting (generating images). Do the subjects reinforce each other, or do they fight over the same mental capacity? Does getting better at language make you a better painter? Does painting practice hurt reading comprehension?

This is exactly the question facing foundation models today. The field is shifting toward unified multimodal pretraining, where a single model learns text, image understanding, and image generation together from the very start. Yet how the modalities actually interact during training has mostly been guided by heuristics. This paper — a collaboration spanning Meta FAIR, Reality Labs, and Oxford — attacks the question the way a physicist would: with tightly controlled experiments that vary one factor at a time. The result is four empirical "laws" of multimodal pretraining plus practical recipes, and the authors validate the conclusions at scale by training multiple 13.5B-parameter MoE models on 2T tokens (§6.2).

The experimental setup: text and image tokens share one Transformer (§2)

The workhorse is a Llama-3-style decoder-only Transformer (1.5B parameters, 2.3B including modality-specific FFNs). Following the Transfusion framework, the model learns text via ordinary next-token prediction and image generation via flow matching — a cousin of diffusion that learns the "flow" carrying noise toward an image — both inside a single model (§2.1).

On the generation side, the network predicts the clean image $x_0$ directly from a noisy input, and that prediction is converted on the fly into a velocity (§2.1):

In plain words: $v$ is the speed and direction needed to travel from the current noisy image $x_t$ to the predicted clean image $x_0$ in the remaining time $1-t$. Here $t$ runs from 0 (pure noise) to 1 (finished image), and at every step the model outputs "which way to move, and how fast."

The crucial design point is that text tokens and image tokens mix inside the same attention mechanism. Attention weights are dot products between queries and keys, so the more the text and image representations "point the same way," the more information flows across the modality boundary. (If queries, keys, and dot products aren't second nature yet, Attention from Scratch builds that foundation.) Keep this dot-product intuition in mind — it underpins the synergy story later.

Finding 1: knowledge flows mostly one way (§3.1)

The paper first measures, on real data (DCLM for language, roughly 350M image–text pairs for vision), what happens to each capability when you scale up one modality's data while holding everything else fixed. The outcome is a strikingly asymmetric picture.

• Language is a universal booster (§3.1.1). With the vision budget fixed, sweeping the language ratio from 0% to 80% monotonically improves every visual-understanding axis (General, Knowledge, OCR & Chart, Vision-Centric), and image generation improves too — diffusion loss drops for both text-conditional and unconditional generation. Learning language alone raises both the model's ability to see and its ability to draw.
• Understanding is a strong teacher for generation (§3.1.2). Adding image-to-text understanding data markedly improves generation metrics and clearly lowers diffusion loss. Pure language performance dips slightly, which the paper attributes to the text distribution in vision-language datasets differing from pure language corpora.
• Generation is roughly neutral (§3.1.3). Scaling up generation data causes only minor fluctuations in language and understanding — no clear boost, no severe damage. Flow-matching objectives don't inherently conflict with next-token prediction, but they transfer little knowledge backward.

So knowledge flows readily downstream — language → understanding → generation — with almost no backflow. This river-like structure is the spine of the whole paper.

Finding 2: whether transfer happens depends on the kind of concept (§3.2)

Real data entangles too many factors, so the paper turns to a controlled synthetic environment built on CLEVR (procedurally rendered scenes of simple objects) and performs surgical ablations: remove one specific concept from exactly one modality's training stream. For example, delete "yellow" from the generation data only, keep it in the understanding data, then ask the model to draw a yellow object (§3.2.1).

The results split cleanly along a conceptual hierarchy (§3.2.2):

• Low-level attributes — color and shape — do not zero-shot transfer in either direction. Remove them from the generation stream and generation collapses; remove them from understanding and VQA on those concepts is destroyed. Basic visual vocabulary must be learned explicitly within each task objective.
• Structural concepts — spatial relations, size, object count — transfer from understanding to generation only. Spatial knowledge learned through understanding helps the model zero-shot generate arrangements it never saw. The reverse direction mostly fails, with counting as the single mild exception.