Paper Walkthrough: DreamX-Creator — Making Sound and Picture Together in 7B, Then Finishing at 2K in One Step

A ground-up walkthrough of a 7B model that denoises audio and video inside one generative process: the gated cross-modal attention, the modality-aware reinforcement learning, the one-step 2K refiner — and the unusually heavy caveats the authors put on their own results.

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

What gets lost when the sound is bolted on afterwards

Almost all the progress in video generation has been about the picture. Audio is either not produced at all, or it is pasted on by a second model once the video is finished.

Why does that hurt? Picture a ball hitting the floor. You believe the impact because the sound lands at the moment of contact. But in a pipeline that goes "make the video, then make audio from that video," the video is finalised without ever knowing about the sound. Audio can be fitted to the picture; the picture can never be fitted to the audio. The relationship is one-way. This paper puts both modalities inside the same generative process and makes it two-way again.

The original title, and what the abstract says

The paper is "DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution" (arXiv:2608.31106, 31 August 2026). Inside the report the system is called DreamX-Creator 1.0.

The abstract runs as follows. Recent video generators omit audio or synthesise it in a separate stage, which limits reciprocal modelling of visual dynamics and acoustic events. The authors present a compact native joint audio-video generation system built around a 7B generator. Conditioned on a first frame and a text prompt, it jointly denoises modality-specialised audio and video streams. The streams run independently through the first half of the network and are coupled in the latter half by Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. Training happens on capability-oriented data pools built by a unified Audio-Video Data System: two audio-video pre-training stages, then High-Quality Finetuning, then reinforcement learning with Modality-Aware Multimodal Feedback, which routes video, audio and cross-modal feedback to the corresponding streams. For high resolution, a bidirectional multi-step teacher is adapted into an autoregressive multi-step refiner and distilled into a student that needs one denoising evaluation per temporal chunk. Releasing the 7B generator and the 2K Refiner is meant to democratize native audio-video generation.

That word "democratizing" maps onto the comparison table (§1, Table 1). Among systems that combine all three of downloadable weights, native joint audio-video generation, and an officially supported output path at 2K or above, this is the smallest one with a disclosed total backbone size — as of publicly documented capabilities on 27 August 2026. The table lines it up against LTX-2.3 (22B), MiniMax H3 (33B), MAGI-2 Preview (114B) and others. The same footnote is careful to say that "Open" means the parameters can be downloaded and does not imply an OSI-approved license. That detail comes back later.

The picture, and the intuition

Think of two booths in a recording studio. In one, someone works on the visuals; in the other, someone works on the sound. Each uses their own tools — their own token rate, their own positional encoding, their own backbone. For the first half of the session they work entirely separately, sharing only the text brief. Halfway through, a window opens between the booths: the video person can look at the audio person's desk (A2V, audio-to-video) and vice versa (V2A, video-to-audio). Two things matter here: whether the window opens at all is switched per sample, and how much of what comes through is actually absorbed is decided per token. The paper implements these as two separate mechanisms — a direction mask and a gate (§3.2).

"Looking through the window" is just cross-attention. Your side issues a Query, the other side holds up Keys and Values, and the larger the dot product between a query and a key, the more strongly that value flows in. The only difference from self-attention is that query and key come from different modalities; the machinery is the same as in attention from scratch. One wrinkle: audio and video have different token rates, so "video token 3" and "audio token 3" do not refer to the same instant. The paper maps both positions onto a shared temporal coordinate system and applies temporal RoPE to the cross-modal queries and keys (§3.1) — no resampling of either latent sequence, just the time correspondence carried in.

Mechanism 1: which stream conditions which is decided by noise level

During training every sample is assigned an A2V, V2A or Joint mode. What is neat is that the mode is expressed purely as the ordering of the two streams' noise levels (§3.1). Writing $\sigma$ for corruption strength:

What that says, in words: leave the conditioning stream relatively clean and corrupt the generated stream harder. Under A2V the video side is the messy one and the audio side stays cleaner, so the model practises "repair a wrecked picture using intact sound as the cue." V2A is the mirror image; Joint corrupts both equally. A stop-gradient sits on the conditioning side, so the target stream's loss cannot rewrite the conditioning backbone through cross-modal attention.

And here is the point the paper keeps hammering: A2V and V2A are not inference tasks. They are internal conditioning paths used during training (§3.1, §3.3). The inference interface is a single one, start to finish: one frame plus a prompt.

Both streams are trained with flow matching (§3.4). From a point interpolated between a clean latent $z_0$ and noise $\epsilon$, the model predicts the velocity $u^*=\epsilon-z_0$ pointing back toward the clean side — see an introduction to flow matching for the groundwork. The total loss is the weighted sum $\lambda_v\mathcal{L}^v+\lambda_a\mathcal{L}^a$, with $\lambda_v=\lambda_a=0.5$ during pre-training and $\lambda_v=0.5,\ \lambda_a=0.1$ during High-Quality Finetuning — the audio term is deliberately turned down at the end.

Mechanism 2: the gate that sets how much gets mixed in, per token and per head

If you simply add the cross-modal output, the carefully specialised representations risk being swamped by the other modality. That is exactly the second of the four challenges the paper lists: the required strength of interaction varies across layers, heads, tokens and samples (§1). So a sigmoid gate is inserted right before the head outputs are combined (§3.2).