Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?

What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget.

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Teaching three subjects to one student

Imagine teaching one student three subjects at once: language, art appreciation (understanding images), and painting (generating images). Do the subjects reinforce each other, or do they fight over the same mental capacity? Does getting better at language make you a better painter? Does painting practice hurt reading comprehension?

This is exactly the question facing foundation models today. The field is shifting toward unified multimodal pretraining, where a single model learns text, image understanding, and image generation together from the very start. Yet how the modalities actually interact during training has mostly been guided by heuristics. This paper — a collaboration spanning Meta FAIR, Reality Labs, and Oxford — attacks the question the way a physicist would: with tightly controlled experiments that vary one factor at a time. The result is four empirical "laws" of multimodal pretraining plus practical recipes, and the authors validate the conclusions at scale by training multiple 13.5B-parameter MoE models on 2T tokens (§6.2).

The experimental setup: text and image tokens share one Transformer (§2)

The workhorse is a Llama-3-style decoder-only Transformer (1.5B parameters, 2.3B including modality-specific FFNs). Following the Transfusion framework, the model learns text via ordinary next-token prediction and image generation via flow matching — a cousin of diffusion that learns the "flow" carrying noise toward an image — both inside a single model (§2.1).

On the generation side, the network predicts the clean image $x_0$ directly from a noisy input, and that prediction is converted on the fly into a velocity (§2.1):

In plain words: $v$ is the speed and direction needed to travel from the current noisy image $x_t$ to the predicted clean image $x_0$ in the remaining time $1-t$. Here $t$ runs from 0 (pure noise) to 1 (finished image), and at every step the model outputs "which way to move, and how fast."

The crucial design point is that text tokens and image tokens mix inside the same attention mechanism. Attention weights are dot products between queries and keys, so the more the text and image representations "point the same way," the more information flows across the modality boundary. (If queries, keys, and dot products aren't second nature yet, Attention from Scratch builds that foundation.) Keep this dot-product intuition in mind — it underpins the synergy story later.

Finding 1: knowledge flows mostly one way (§3.1)

The paper first measures, on real data (DCLM for language, roughly 350M image–text pairs for vision), what happens to each capability when you scale up one modality's data while holding everything else fixed. The outcome is a strikingly asymmetric picture.

• Language is a universal booster (§3.1.1). With the vision budget fixed, sweeping the language ratio from 0% to 80% monotonically improves every visual-understanding axis (General, Knowledge, OCR & Chart, Vision-Centric), and image generation improves too — diffusion loss drops for both text-conditional and unconditional generation. Learning language alone raises both the model's ability to see and its ability to draw.
• Understanding is a strong teacher for generation (§3.1.2). Adding image-to-text understanding data markedly improves generation metrics and clearly lowers diffusion loss. Pure language performance dips slightly, which the paper attributes to the text distribution in vision-language datasets differing from pure language corpora.
• Generation is roughly neutral (§3.1.3). Scaling up generation data causes only minor fluctuations in language and understanding — no clear boost, no severe damage. Flow-matching objectives don't inherently conflict with next-token prediction, but they transfer little knowledge backward.

So knowledge flows readily downstream — language → understanding → generation — with almost no backflow. This river-like structure is the spine of the whole paper.

Finding 2: whether transfer happens depends on the kind of concept (§3.2)

Real data entangles too many factors, so the paper turns to a controlled synthetic environment built on CLEVR (procedurally rendered scenes of simple objects) and performs surgical ablations: remove one specific concept from exactly one modality's training stream. For example, delete "yellow" from the generation data only, keep it in the understanding data, then ask the model to draw a yellow object (§3.2.1).

The results split cleanly along a conceptual hierarchy (§3.2.2):

• Low-level attributes — color and shape — do not zero-shot transfer in either direction. Remove them from the generation stream and generation collapses; remove them from understanding and VQA on those concepts is destroyed. Basic visual vocabulary must be learned explicitly within each task objective.
• Structural concepts — spatial relations, size, object count — transfer from understanding to generation only. Spatial knowledge learned through understanding helps the model zero-shot generate arrangements it never saw. The reverse direction mostly fails, with counting as the single mild exception.

The follow-up experiment is the most interesting part (§3.2.3). Even the colors and shapes that failed to transfer zero-shot leave something behind: when the missing concept is later reintroduced via fine-tuning, prior exposure through generation dramatically accelerates learning it for understanding (a gain in mean accuracy between +0.133 and +0.273). The generative objective builds dense, pixel-level representations that sit dormant as a latent prior until the model is shown how to map them to language. The abstract features learned through understanding, by contrast, provide almost no such foundation for generation. The paper uses this to reconcile conflicting prior reports: generative pretraining tends to help understanding only on spatial, geometric, and physical reasoning tasks — precisely the tasks that need dense features.

Finding 3: data complexity decides synergy vs. competition (§4.1)

Next question: when do modalities help each other, and when do they fight over capacity? The paper varies the complexity of one modality's data while freezing the other, and the answer is counterintuitive.

Simple tasks boost the other modality. Pair language training with extremely simple images — solid-color backgrounds — and language perplexity beats a pure language model trained without vision at all (Δppl −0.211). Symmetrically, pairing vision with a maximally simplified language stream gives the biggest reduction in generation loss. But escalate the visual stream toward complex real-world distributions like video and natural photos, and synergy flips into competition: text perplexity degrades (Δppl +0.075 for natural photos). Synergy and capacity competition coexist in the same training loop, and as tasks get harder, competition wins — that is the dynamic the paper maps out.

What to share, what to split (§4.2–4.3)

So where inside the network does synergy live, and where does competition bite? The paper deconstructs the Transformer block into three parameter groups — FFN, attention, and normalization — and tests shared vs. modality-specific versions of each (§4.2):

• Fully shared (dense) is the worst. Both modalities grind against the same weights, and both end up worse than unimodal baselines (ppl +0.272, diffusion loss +0.0537).
• Splitting only the FFNs (split_ffn) is the best. Competition disappears, and shared attention channels the synergy: both modalities beat their unimodal baselines (ppl improved by 0.211, vision loss by 0.0168).
• Splitting attention or normalization kills the synergy. Fully split networks perform essentially like unimodal baselines — unification stops meaning anything.

The paper's reading: shared attention and normalization are the bridge that carries synergy; separate FFNs are the pressure valve that absorbs competition. This is direct empirical support for Mixture-of-Transformers and MoE designs with modality-specific experts. And the pattern is robust: it persists across four different visual tokenizer designs — semantic-latent RAE, raw pixels, CLIP + VAE, and discrete autoregressive codes (§4.3). Synergy comes from the training objectives and the parameter-sharing design, not from any particular visual representation.

Finding 4: integrate vision late and it gets lazy (§5)

If language is a universal booster, it's tempting to train a pure language model first and bolt vision on afterward — which is exactly what most production pipelines have done. The paper argues this is a mistake.

Fixing the total budget at 1T tokens and sweeping only the moment vision enters (after 0 to 800B pure-language tokens), the language gains plateau beyond roughly 600B, while nearly every visual understanding and generation metric declines steadily the later vision is introduced (§5.1). The paper also tests all six sequential orderings of language, understanding, and generation — with and without a 12.5% replay buffer against forgetting — and not one sequential curriculum matches simultaneous joint training (§5.2).

Why? Dissecting checkpoints that differ only in the length of the language warm-up, the paper identifies a phenomenon it calls vision laziness (§5.3). The longer the language-only phase, the more (A) the vision-specific FFNs' training-time activations go quiet across layers, (B) the embeddings of the special image-wrapper tokens get pushed to the margins of the language space, (C) the vision FFNs' inference-time output collapses, and (D) the share of attention actually directed at image tokens falls. A hardened language trunk lets the model "shortcut" visual problems linguistically, so it never bothers to fully optimize its visual pathway. The paper suggests this may be one reason even powerful multimodal LLMs occasionally act blind on elementary visual details.

The recipe: a 70/25/5 mix (§6)

Put the laws together and a practical recipe falls out. If knowledge flows language → understanding → generation and never back, then most of the budget should go to language and understanding, with generation needing only a sliver. An extensive grid search over 1T-token runs finds the optimum at 70% language / 25% understanding / 5% generation (§6.1). Cutting generation data to just 5% barely hurts generation quality — and pushes the understanding average to its best value in the sweep.

The full recipe — asymmetric mix, split-FFN-style MoE, early unification — is then validated at 13.5B MoE scale (1.5B active parameters, 256 experts with one fixed language expert and one fixed vision expert) on 2T tokens, against baselines that each change exactly one variable (§6.2). The asymmetric mix beats the balanced 50/25/25 recipe across the board: language accuracy 54.31% vs. 52.86%, visual understanding average 43.08% vs. 41.42%, GenEval 0.482 vs. 0.467 — while using five times fewer generation tokens (image quality stays essentially even, FID 5.234 vs. 5.131). The MoE-vs-dense and early-vs-late comparisons likewise reproduce the small-scale conclusions at scale.

Limitations the paper itself flags (Appendix A)

Two boundaries worth keeping in mind. First, the experiments cover text and static images only; whether the same laws hold for video, audio, and other time-based modalities is untested. Second, validation stops at 13.5B parameters and 2T tokens. At frontier scale (1T+ parameters), the synergy–competition threshold could shift, and the paper explicitly leaves open the possibility that the currently absent generation-to-understanding backflow could emerge when a huge model can use generation as an internal world simulator. The 5% generation ratio is also an optimum for this specific setup — the authors note that raising the generation share later in training, or adding a dedicated generation mid-training stage, can still help (§6.2). Read every number as tied to this paper's experimental system (Transfusion-style training, its particular datasets), not as universal constants.

How this gets used in practice

The people this paper speaks to most directly are ML engineers and researchers designing multimodal pretraining runs. The knobs it informs are the data loader's modality mixing ratios (the L/U/G split here — a balanced mix likely wastes generation tokens), the loss weighting (the paper up-weights the diffusion loss 3× relative to text cross-entropy (§2.1)), and the parameter-sharing layout (share attention and normalization; isolate FFNs per modality, or pin modality-specific experts in an MoE).

Even if you never pretrain from scratch, the findings travel. If you fine-tune vision onto an existing LLM (LLaVA-style pipelines), vision laziness becomes a diagnostic checklist: log the activation norms of the vision-side modules and the fraction of attention landing on image tokens. If those numbers are sinking, your model is likely shortcutting through language priors instead of looking at the image. Evaluation can mirror the paper's public benchmarks — GenEval, DPG-Bench, CLIP similarity.

Two traps to avoid. (1) "More generation data will improve understanding" contradicts the measured direction of knowledge flow — the exception, per §3.2.3, is if what you care about is spatial/geometric reasoning, where generation-learned dense features can be activated by fine-tuning. (2) "A stronger language model is always a better multimodal base" is not guaranteed — the longer the language-only warm-up, the lazier the vision pathway, so consider integrating from an earlier checkpoint or unifying from scratch. And if an interviewer asks "do understanding and generation help each other?", the answer this paper equips you with is: transfer is asymmetric and concept-dependent — structural concepts flow from understanding to generation, and almost nothing flows back zero-shot.

Takeaways

• Knowledge flows asymmetrically — language → understanding → generation — with almost no backflow (§3)
• Synergy is strongest with simple data and gives way to capacity competition as complexity rises; shared attention/norm plus split FFNs is the winning layout (§4)
• The later vision is introduced, the lazier the vision pathway gets; no sequential curriculum beats joint training (§5)
• The L70/U25/G5 mix with MoE and early unification holds up at 13.5B parameters and 2T tokens (§6)

For the foundations behind this paper, see our primers on diffusion models, Mixture of Experts (the scaled-up cousin of the split-FFN design), and CLIP, which pioneered aligning text and image representations.