Paper Explained: JoyAI-Video-Edit — Autoregressive Diffusion That Keeps Editing 720p Video at ~30 FPS, Without Ever Seeing the Future

Say 'swap the background for snowy mountains' and the live feed comes back edited as it streams — a walkthrough of JD's JoyAI-Video-Edit from the paper itself. We trace the three-stage recipe (chunk-wise autoregression, Source-Anchored DMD, long-horizon distillation) that lets a 16B model edit 720p video at roughly 30 FPS.

JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

In one sentence

JoyAI-Video-Edit (arXiv:2608.03974, from JD's Joy Future Academy) is a 16B-parameter autoregressive diffusion model that keeps editing an incoming video stream according to your instruction without ever looking at future frames. The video's length does not need to be known in advance (it is "open-ended"), and the full system edits 720p footage at roughly 30 FPS on a single Nvidia B200 GPU (Abstract, §5). The code is public on GitHub.

An analogy: translation vs. simultaneous interpreting

Conventional video-editing AI works like a book translator: it reads the whole clip before writing the edited result. Quality is high, but nothing comes out until the end. Editing a live stream is simultaneous interpreting: the speaker is still talking, nobody knows when they will stop, you must translate as the words arrive — and keep the tone you started with all the way through. The paper formalizes these requirements as causal output (no peeking at the future), low latency, computation and memory that do not grow with video length, and stable quality over an unknown duration (§1).

Why the existing playbook fails

Most high-quality editors are offline by design: they apply bidirectional attention (looking at both past and future) over the entire clip and only emit the result after iterative denoising finishes. Depending on future frames rules out causal output, and memory grows with input length. The obvious shortcut — chop a long video into clips and run an offline editor on each — is also rejected by the paper: it recomputes overlapping context and produces visible discontinuities at clip boundaries (§1).

Merely making the model causal is not enough either. An autoregressive model trains on clean ground-truth history but, at inference time, must keep consuming its own imperfect outputs as history. This train–inference mismatch lets color deviations and reconstruction errors propagate through the generated history and accumulate into long-term drift (§1). Editing makes things harder still: every output chunk must stay aligned with the source chunk that just arrived, preserve everything the instruction did not target (identity, motion, background), and apply the requested transformation consistently — while the few-step generation needed for real-time throughput opens yet another gap (§1).

Three building blocks (§3.1)

The architecture has three parts: (1) an MLLM reads the source video's first frame plus the editing instruction and produces condition tokens encoding "what to change and how"; (2) a causal video VAE squeezes the video into latent space at an 8×24×24 compression ratio (one latent frame corresponds to eight real frames); (3) an MM-DiT diffusion transformer generates the edited latents from the condition and latent tokens. It supports both instruction-only V2V and reference-image IV2V editing.

Foundation training (§3.2) is a progressive curriculum — text-to-image, text-to-video, image editing, then bidirectional video editing — with a flow-matching objective. (The add-noise-then-remove-it groundwork underneath all of this is covered in Diffusion Models from the Ground Up.)

Here $z_0$ is the latent of the edited target video (the answer), $\epsilon$ is Gaussian noise, and $\sigma$ is a noise level between 0 and 1. The left equation mixes answer and noise; the model practices predicting $v^{\star}$ on the right — the direction that carries the mixture back to the answer (§3.2, Eqs. (1)–(2)). Spelled out, the pair is a recipe which says: stir the finished edit together with noise in a ratio you control, then have the model name the direction that undoes the stirring.

Step 1: chunk-wise autoregression (§4.1)

First, the bidirectional editor is converted to generate chunk by chunk, front to back. The video is cut into fixed-size chunks, with bidirectional attention inside each chunk and causal attention across chunks. Instead of attending to the entire past, a sliding window keeps only the most recent chunks — with one exception: the very first chunk is retained forever as a global sink, a persistent anchor. However long the stream grows, per-chunk compute and memory stay constant (§4.1).