4DAnyone, Explained — Turning One Casual Video Into a 4D Person

How 4DAnyone builds a free-viewpoint 4D human from a single phone video, explained from scratch. The core trick is not a better generator but two fixes — RCP and TCR — for a context that no longer fits in one attention pass.

4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Seeing someone from an angle you never filmed

You film a friend playing bass, from the front, on your phone. Later you want to see the back of their hands. You can't — the angles you didn't shoot simply aren't there.

4DAnyone attacks exactly that. In the paper's terms, it reconstructs 4D humans from an uncalibrated monocular video (an ordinary handheld clip with unknown intrinsics and poses) by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (Abstract). What makes it interesting is the claim that raw generation quality is not the hard part — staying consistent across dozens of views is.

4D means 3D plus time

Let's fix the vocabulary. 3D is a shape you can orbit; 4D is a shape you can orbit while it moves. A statue is 3D; a dancer is 4D.

The dominant way to represent that today is Gaussian Splatting. Instead of building the subject from polygons, you float hundreds of thousands of small translucent ellipsoidal blobs in space, each with its own colour, size and orientation, and composite them onto the screen — painting the world in fog rather than in triangles. Let the blobs move over time and you get 4D Gaussian Splatting (4DGS); this paper uses FreeTimeGS as its downstream reconstructor (§3.1).

The catch is that 4DGS demands expensive inputs. The DNA-Rendering capture the paper cites uses a 48-camera rig of calibrated, synchronised, static cameras (§1). No studio, no reconstruction. Hence the idea: if multiview footage is what 4DGS wants, generate it from a single video.

Why the naive route breaks

The obvious plan: ask a camera-controlled video diffusion model for "this person seen from the right rear," repeat for sixteen viewpoints, feed the pile to 4DGS.

The paper reports that existing camera-controlled methods fall apart at this scale (§1). Individually the clips look plausible. Push to the tens of views 4DGS needs and the shirt pattern shifts between views while the silhouette quietly gains and loses weight. A human eye may not notice; 4DGS certainly does, because it resolves geometry from cross-view agreement, so small disagreements become broken reconstructions. The paper names the symptoms appearance inconsistency and structural drift, and concludes that the bottleneck "is not only view control, but how to maintain cross-view consistency at reconstruction scale."

The culprit: attention context is a finite container

Why does scale break it? The diagnosis is architectural rather than about model capacity: the attention context of a single DiT forward pass is bounded by practical memory and compute budgets (§1).

One line of refresher on attention: every token compares itself to every other token and pulls in information from whichever ones it resembles, where resemblance is a dot product (see Attention From Scratch). 4DAnyone adds multiview self-attention, which rearranges tokens so different viewpoints at the same timestep attend to each other directly; it reuses the architecture and the weights of the base Wan2.2 DiT's temporal self-attention (Supp. A).

Which means only views sitting in the same container can negotiate with each other. Sixteen views times 121 frames of tokens do not fit, so views must be split into groups — and that split creates two bottlenecks (§1).

The first is on the reference side. Ideally each group would condition on every view generated so far. But that reference context grows as $O(N)$ in the number of views and quickly overruns the budget; trim it to fit and you starve the model of appearance guidance.

The second is on the target side. Groups denoised separately have disjoint contexts, so they cannot exchange "here is the body shape and posture we've settled on," and structure drifts apart group by group. No prior method solves both at once, the paper argues: CAT3D picks sparse anchor views as context per batch but throws away the appearance information of everything unselected, while CAT4D and Diffuman4D use sliding-window denoising with overlap aggregation, where cross-window drift survives at large view counts (§2).

Fix 1: Reference Context Packing — folding references down to $O(1)$

Reference Context Packing (RCP) kills the $O(N)$ on the reference side. The observation is that nearby viewpoints carry heavily redundant appearance information, so a mixed-resolution reference context suffices to preserve global layout while retaining fine detail (§3.3).

The mechanism is almost anticlimactic. Video diffusion models cut frames into patches to make tokens — the layer that does this is the patchify layer, and in base Wan2.2 it has kernel and stride $(1,2,2)$. RCP adds a layer $\mathcal{P}_r$ with kernel and stride $(1,2r,2r)$: coarser patches by a factor of $r$ per side, so $\frac{1}{r^2}$ as many tokens. Concretely the $2\times$ and $4\times$ layers use $(1,4,4)$ and $(1,8,8)$ (Supp. A). The reference context actually assembled is:

In words: keep the source video $\mathbf{V}_{\text{src}}$ at full resolution, three already-generated references at $2\times$ compression, and four more at $4\times$. Counting tokens, that is $1+3\cdot\frac14+4\cdot\frac{1}{16}=2$ — eight references seated in two full-resolution seats' worth of space. The number of seats is fixed, so reference-context complexity drops from $O(N)$ to $O(1)$.