4DAnyone, Explained — Turning One Casual Video Into a 4D Person

How 4DAnyone builds a free-viewpoint 4D human from a single phone video, explained from scratch. The core trick is not a better generator but two fixes — RCP and TCR — for a context that no longer fits in one attention pass.

4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Seeing someone from an angle you never filmed

You film a friend playing bass, from the front, on your phone. Later you want to see the back of their hands. You can't — the angles you didn't shoot simply aren't there.

4DAnyone attacks exactly that. In the paper's terms, it reconstructs 4D humans from an uncalibrated monocular video (an ordinary handheld clip with unknown intrinsics and poses) by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (Abstract). What makes it interesting is the claim that raw generation quality is not the hard part — staying consistent across dozens of views is.

4D means 3D plus time

Let's fix the vocabulary. 3D is a shape you can orbit; 4D is a shape you can orbit while it moves. A statue is 3D; a dancer is 4D.

The dominant way to represent that today is Gaussian Splatting. Instead of building the subject from polygons, you float hundreds of thousands of small translucent ellipsoidal blobs in space, each with its own colour, size and orientation, and composite them onto the screen — painting the world in fog rather than in triangles. Let the blobs move over time and you get 4D Gaussian Splatting (4DGS); this paper uses FreeTimeGS as its downstream reconstructor (§3.1).

The catch is that 4DGS demands expensive inputs. The DNA-Rendering capture the paper cites uses a 48-camera rig of calibrated, synchronised, static cameras (§1). No studio, no reconstruction. Hence the idea: if multiview footage is what 4DGS wants, generate it from a single video.

Why the naive route breaks

The obvious plan: ask a camera-controlled video diffusion model for "this person seen from the right rear," repeat for sixteen viewpoints, feed the pile to 4DGS.

The paper reports that existing camera-controlled methods fall apart at this scale (§1). Individually the clips look plausible. Push to the tens of views 4DGS needs and the shirt pattern shifts between views while the silhouette quietly gains and loses weight. A human eye may not notice; 4DGS certainly does, because it resolves geometry from cross-view agreement, so small disagreements become broken reconstructions. The paper names the symptoms appearance inconsistency and structural drift, and concludes that the bottleneck "is not only view control, but how to maintain cross-view consistency at reconstruction scale."

The culprit: attention context is a finite container

Why does scale break it? The diagnosis is architectural rather than about model capacity: the attention context of a single DiT forward pass is bounded by practical memory and compute budgets (§1).

One line of refresher on attention: every token compares itself to every other token and pulls in information from whichever ones it resembles, where resemblance is a dot product (see Attention From Scratch). 4DAnyone adds multiview self-attention, which rearranges tokens so different viewpoints at the same timestep attend to each other directly; it reuses the architecture and the weights of the base Wan2.2 DiT's temporal self-attention (Supp. A).

Which means only views sitting in the same container can negotiate with each other. Sixteen views times 121 frames of tokens do not fit, so views must be split into groups — and that split creates two bottlenecks (§1).

The first is on the reference side. Ideally each group would condition on every view generated so far. But that reference context grows as $O(N)$ in the number of views and quickly overruns the budget; trim it to fit and you starve the model of appearance guidance.

The second is on the target side. Groups denoised separately have disjoint contexts, so they cannot exchange "here is the body shape and posture we've settled on," and structure drifts apart group by group. No prior method solves both at once, the paper argues: CAT3D picks sparse anchor views as context per batch but throws away the appearance information of everything unselected, while CAT4D and Diffuman4D use sliding-window denoising with overlap aggregation, where cross-window drift survives at large view counts (§2).

Fix 1: Reference Context Packing — folding references down to $O(1)$

Reference Context Packing (RCP) kills the $O(N)$ on the reference side. The observation is that nearby viewpoints carry heavily redundant appearance information, so a mixed-resolution reference context suffices to preserve global layout while retaining fine detail (§3.3).

The mechanism is almost anticlimactic. Video diffusion models cut frames into patches to make tokens — the layer that does this is the patchify layer, and in base Wan2.2 it has kernel and stride $(1,2,2)$. RCP adds a layer $\mathcal{P}_r$ with kernel and stride $(1,2r,2r)$: coarser patches by a factor of $r$ per side, so $\frac{1}{r^2}$ as many tokens. Concretely the $2\times$ and $4\times$ layers use $(1,4,4)$ and $(1,8,8)$ (Supp. A). The reference context actually assembled is:

In words: keep the source video $\mathbf{V}_{\text{src}}$ at full resolution, three already-generated references at $2\times$ compression, and four more at $4\times$. Counting tokens, that is $1+3\cdot\frac14+4\cdot\frac{1}{16}=2$ — eight references seated in two full-resolution seats' worth of space. The number of seats is fixed, so reference-context complexity drops from $O(N)$ to $O(1)$.

Inference proceeds progressively. References are generated first (§3.3 describes two rounds; the evaluation setup generates four references in a single round — Supp. F), with viewpoints chosen by farthest-point sampling: greedily add the candidate whose minimum angular distance to already-selected views is largest, which spreads coverage instead of clustering it. Once the references exist, $\mathcal{C}_{\text{R}}$ is frozen and every target group shares it. During training, $\mathcal{C}_{\text{R}}$ gets dropout with probability 0.1 on each of two blocks, which is what lets the model run in the very first round when no references exist yet (§3.3).

Fix 2: Target Context Routing — rotating the groups

The target side — groups that cannot talk to each other — is closed by Target Context Routing (TCR), without spending extra memory.

The lever is the temporal structure of diffusion: high-noise steps establish global structure, low-noise steps refine local detail (§3.3). If the way a diffusion model climbs from noise to image is unfamiliar, Diffusion Models: An Introduction covers it. So change the grouping over time. Past a switching timestep $t_s$, the high-noise phase cyclically shifts the ordered view indices by the step index before re-partitioning into four-view groups — who you share a container with changes every step, so structure propagates even between views never grouped together at any single step. The low-noise phase fixes adjacent four-view groups so neighbouring views refine detail together and cross-view transitions stay stable.

views = list(range(16))                 # target views ordered by azimuth
for n in range(T, 0, -1):               # T = 20 denoising steps
    if t[n] > t_s:                      # high noise: slide the ordering
        s = n % 16
        order = views[s:] + views[:s]
    else:                               # low noise: fixed adjacent groups
        order = views
    for g in [order[i:i+4] for i in range(0, 16, 4)]:
        z[g] = denoise(z[g], ref=C_R, skel=S[g], t=t[n])

The per-group budget never changes; only the routes information can take.

Geometry: accuracy over density

The other design decision is what geometric hint to condition on. Explicit prior methods lean on dense signals such as depth maps, but the paper argues these are hard to estimate reliably from in-the-wild video, and that errors become conflicting geometric constraints that make multiview generation diverge (§3.2).

Hence accuracy over density: use 3D skeletons instead, since modern human mesh recovery pulls joint positions out of monocular video robustly. GVHMR estimates a ground-aligned SMPL-X mesh sequence, and a sparse vertex-to-keypoint regressor extracts 70 Goliath-vocabulary keypoints; on held-out scenes that regressor reaches 3.5 mm mean keypoint error against 14.0 mm for a nearest-vertex baseline (Supp. D).

Rendering the skeleton takes one more idea. A plain 2D skeleton drawing is front-back ambiguous: an arm in front of the torso and an arm behind it produce the same picture, which yields contradictory results across viewpoints. The paper therefore rasterises the 3D skeleton with a pixelwise z-buffer so nearer parts occlude farther ones (§3.2) — occlusion awareness without adding an input channel. The keypoint set is deliberately small: 40 points selected from the 308-point Goliath vocabulary (17 body, 6 foot, 10 palm-level hand, 7 auxiliary), with the 238 facial keypoints and 30 finger joints excluded on purpose. Expressions and finger detail are learned from the source video instead, avoiding artefacts from noisy fine-grained detections. The skeleton enters as a residual on the noisy latents:

which says: take the noisy latent token $\mathbf{z}^{t}_{i}$ for view $i$ at timestep $t$ and add the encoding of that view's depth-buffered skeleton video $\mathbf{S}_{i}$. The final projection layer of $g_{\phi}$ is zero-initialised, so the residual starts at zero and the pretrained DiT behaviour is preserved when training begins.

Training

The data is five sources (Table 1): MVGameHuman, built with the authors' in-house game engine (38k videos, 24 cameras, 318 actors), plus SynCamVideo and the real light-stage DNA-Rendering (51k videos, 48 cameras), topped up with monocular TedTalk and Pexels. Training is a three-stage curriculum: stage 1 learns skeleton-conditioned camera control on foreground-only DNA-Rendering (with probability 20% the source and target clips come from different temporal windows, forcing the model to follow the skeleton for pose while borrowing appearance from the source); stage 2 drops masking and adds all multiview data; stage 3 adds monocular data and removes finger keypoints (§3.4). The objective is a latent flow-matching loss — the framework covered in Flow Matching: An Introduction — plus an LPIPS term weighted $\lambda=0.25$. The base model is Wan2.2-TI2V-5B, fine-tuned at 704×1280 on 128 H20-3E GPUs for about three days across the three stages; inference uses 20 denoising steps, four-view groups, and $t_s/T=0.2$ (§4.1).

What the numbers say

Evaluation covers 10 DNA-Rendering scenes and 3 DyMVHumans scenes, none seen during training, with 16 cameras and 98 frames each, generating 16 target views from one front-view source (§4.1). PSNR on DNA-Rendering, higher is better (Table 2):

"Generated video consistency" holds out 4 views, builds 4DGS from the other 12, and compares renderings at the held-out views against the generated videos — a measure of 3D agreement among the generations themselves. On DyMVHumans, out of distribution for every method, 4DAnyone likewise leads with 24.48, 23.28 and 21.03. Note the dagger: ReCamMaster† is the authors' own fine-tune, trained on the same data with the same protocol and equipped with the same RCP module and TCR strategy. The comparison is therefore implicit camera-parameter conditioning versus explicit skeleton-geometry conditioning, not tuned versus untuned.

The ablation is equally readable (Table 3, generated video consistency):

The last three rows carry the lesson. As a way of rotating groups, a random permutation gives no measurable gain over fixed grouping, and a strided reordering degrades every metric. Only Sliding — shifting the ordering one position at a time — improves all three, and the paper's reading is that preserving local view adjacency matters when routing (§4.3). Shuffling for its own sake does not work. The switching-time sweep agrees (Table 7): PSNR climbs from 22.2079 at $t_s/T=1$ (fixed grouping throughout) to 22.6294 at $t_s/T=0.2$, then saturates, reaching 22.6414 at $t_s/T=0$; 0.2 is chosen as the entry point of that plateau. Costs are modest (Supp. E): preprocessing about 2 minutes on one RTX 4090, multiview generation about 7 minutes on one H20, and FreeTimeGS 4DGS training about 30 minutes on one RTX 4090.

Where it stops working

The paper states its limits plainly (Supp. H). First, loose garments: a skeleton says nothing about fabric that swings far from the body, so large flowing cloth is generated inconsistently across views and drags the reconstruction down. The very choice that makes the method robust also caps it. Second, pose estimation errors: generation follows the given skeleton faithfully, so a wrong skeleton is inherited by every view. In the paper's example a dancer stands en pointe, HMR predicts flat feet, and all generated views reproduce the error; downstream 4DGS cannot repair it. The flip side, which the paper also notes, is that even under occlusion or motion blur HMR usually returns a complete, plausible skeleton, so pose errors give a slightly shifted but coherent human rather than a broken one.

As a reader's caveat: these are human-centric benchmarks with relatively simple backgrounds. The paper gives no numbers for multiple interacting people or for large props moving with the subject, so treat those as untested, and note that while Table 6 lists 16-, 32- and 48-camera configurations, every quantitative result is at 16 views. Ethics gets its own section: the authors acknowledge risks of deepfake misuse, identity privacy violation and copyright infringement, and ask that outputs be disclosed as synthetic and produced only with the consent of the subject.

How this gets used in practice

The people who touch this are virtual production and XR content teams, sports and dance video analysts, and engineers on e-commerce or avatar 3D pipelines. It pays off when you need free-viewpoint material but cannot book a multi-camera stage — for instance, re-framing footage already shot on a single camera.

The knobs you actually turn: the number of target views (16 / 32 / 48 — one layer of 16 suffices for free-viewpoint rendering at limited pitch angles, two layers of 32 covers most scenarios, three layers of 48 is for complex clothing or extreme motion); the TCR switching time $t_s/T$ (default 0.2 — raise it toward fixed grouping and consistency drops); denoising steps (20, or 10 if you can trade a little quality); the RCP compression ratio $r\in\{2,4\}$; and downstream, FreeTimeGS at 50k iterations with learning rate $1.6\times10^{-4}$. Upstream, HMR is GVHMR and 2D keypoint detection is Sapiens2-1B.

Three pitfalls. First, upstream pose errors propagate to every view — the thing to inspect first is not the generated imagery but the skeleton. En pointe, deep squats, hands on the floor: atypical poses are where HMR slips, and one slip is baked into all sixteen views. Eyeballing the skeleton rendering before launching generation saves a lot of GPU time. Second, wardrobe decides your yield: long skirts and capes are a failure mode the paper names explicitly, so on shoots you control, body-following clothing changes the outcome more than any hyperparameter. Third, "denser geometry must be more accurate" is a trap: TrajectoryCrafter, which depends on warped depth, collapses under front-to-back viewpoint changes and trails badly on PSNR (Table 2). For in-the-wild inputs a sparse but accurate signal beats a dense but noisy one — a judgement worth carrying into other generation pipelines.

If this comes up in a design review, the question will be why consistency breaks when you add views. The path to a good answer: it is not model capacity but the finite attention context of one forward pass; that forces grouping; grouping produces two symptoms, $O(N)$ growth on the reference side and disconnection on the target side; and the fix is two-part — fold the references to a constant (RCP) and rotate the groupings over denoising time (TCR).

Wrapping up

4DAnyone's contribution is not a new generative model but context engineering that lifts an existing video diffusion model to reconstruction-grade precision. It takes the mundane constraint that attention containers are finite and answers it on both sides: compress the reference context to $O(1)$ at mixed resolution, and route target groups over time so information can cross. On top of that it picks sparse, accurate 3D skeletons over dense, uncertain depth. None of it is flashy — but it beats prior methods on all three evaluation axes across both benchmarks.