Paper walkthrough: WanPE — turning a video prompt into a director's shooting plan

WanPE rewrites a one-line request into a shot-by-shot plan covering camera, lighting and sound. We explain, from zero, why it builds training pairs backwards from real videos and how SC-GRPO penalizes dropped requirements.

WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

What this paper does

This walkthrough covers "WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation" (arXiv:2609.30221), from Alibaba's Wan Team and collaborators. The subject is the prompt enhancer: the model that sits in front of a video generator and rewrites what the user typed.

The abstract, in short: video generation starts in text, by writing a cinematic screenplay, and only then becomes pixels. Now that generators reach 30 seconds and follow complex conditions faithfully, the prompt effectively directs the production, planning how action, camera trajectories, lighting and sound unfold across multiple shots. The authors present WanPE, a 397B-parameter enhancer trained on 1.05M real-world videos. It learns shot-level cinematic plans through "video-grounded reverse construction" and uses Semantic-Consistency GRPO (SC-GRPO) to keep user requirements intact across shots and time. On WanPEval, a human-annotated 5–30-second testbed backed by roughly 11K blind pairwise judgments, pairing WanPE with Wan3.0's generator raises human preference over raw prompts by 10.66–18.84 points at 5–15 seconds and by 50.86 points at 30 seconds. Reverse construction beats forward rewriting, WanPE leads every commercial system evaluated at 5–15 seconds, and it stays competitive with Seedance 2.5 at 30 seconds.

An analogy: the one-liner and the shooting script

A producer says, "Ten seconds: a young man fights an evil spirit in an alley on a rainy night." No director can shoot from that. How many cuts? Wide or close? Handheld or locked off? Is the rain or the scream on top of the mix? The document that settles all of this is the shooting script.

Video generators have the same split. Something has to turn the short request $x$ into the detailed condition $y$ the generator reads. According to the paper, earlier generators (Wan2.2 is the example) had short context and weak instruction following, so enhancement was little more than cosmetic description (§1). Now that generators handle 30-second, multi-shot video, the enhancer has been promoted to director.

The intuition: why build the data backwards

Training an enhancer needs lots of "request → good script" pairs. The obvious recipe is to hand each request to an LLM and ask it to add camera, lighting and so on. The paper calls this forward rewriting.

The catch is that the resulting scripts carry the habits of an LLM imagining a film. The generator, meanwhile, was trained on captions of real videos, so what it sees at inference drifts away from what it saw in training (§1). A professionally shot fight scene, by contrast, already contains the coordination between acting, cinematography, editing, lighting and sound, fully realized.

So WanPE flips the direction: caption a real video into a script first, then write a plain request that the script would satisfy (§2.2). Secure the answer first, invent the question afterwards.

Mechanism 1: writing down the goal

The system is a three-stage relay (§2.1).

In words: the user writes a request $x$, the enhancer $\pi_\theta$ turns it into a script $y$, and the generator $G_\phi$ renders a video $\hat v$. The ideal script distribution is:

Here $p_{\mathrm{vg}}$ measures how much a text looks like a caption of a real video, $\mathbb{I}[\cdot]$ is 1 when its condition holds and 0 otherwise, $\mathcal{Y}_{\mathrm{sem}}(x)$ is the set of scripts that drop, alter and contradict none of the requirements, and $Z(x)$ rescales everything to sum to one. In words, the ideal is "among texts that read like real captions, keep only those that honor every request." Training then minimizes the KL divergence between this target and the enhancer (see KL divergence from scratch). Reverse-constructed SFT supplies the "real-caption" half; SC-GRPO supplies the "honor everything" half.