4DAnyone: Create Anyone in 4D from a Casual Monocular Video

arXiv:2608.203352026-08-19

Turning a single handheld video into a walk-around 4D human you can view from any angle

4DAnyone takes one ordinary phone video of a person and generates dozens of consistent virtual-camera views of that same person, then turns those views into a 4D Gaussian Splatting model that can be rendered from any viewpoint at any moment in time. The key problem it solves is that AI video generators start to contradict themselves (different body shapes, drifting clothing) once you ask for the tens of views needed for real 3D reconstruction. The authors fix this with two new tricks, Reference Context Packing and Target Context Routing, plus a synthetic training dataset called MVGameHuman, and show it beats prior methods on two benchmark datasets.

METAL MEDIA explanatory visual

Turning a single handheld video into a walk-around 4D human you can view from any angle

01Problem: existing camera-controlled video AI models look fine for a few novel views but fall apart in consistency once scaled to the ~16+ views needed to reconstruct a full 3D human, because the AI can only 'attend' to a limited context at once.
02Reference Context Packing (RCP) compresses all previously generated reference views into a fixed-size, mixed-resolution memory instead of letting it grow indefinitely, keeping computation cost constant regardless of how many views have been generated.
03Target Context Routing (TCR) rotates which views are grouped together during the generation process: views are shuffled together early (high-noise steps) to agree on overall body structure, then locked into fixed neighboring groups later (low-noise steps) to polish fine details.
043D-aware skeleton conditioning: instead of dense depth maps (hard to estimate reliably), the system uses accurate but sparse 3D body skeletons rendered with proper depth ordering, resolving front/back ambiguity that flat 2D skeletons cannot.
05On DNA-Rendering and DyMVHumans test datasets, 4DAnyone outperformed baseline methods (MV-Performer, TrajectoryCrafter, and a fine-tuned ReCamMaster) in generated video quality, cross-view consistency, and final 4D reconstruction quality, and generalized well to casual in-the-wild videos.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Problem: existing camera-controlled video AI models look fine for a few novel views but fall apart in consistency once scaled to the ~16+ views needed to reconstruct a full 3D human, because the AI can only 'attend' to a limited context at once.
Reference Context Packing (RCP) compresses all previously generated reference views into a fixed-size, mixed-resolution memory instead of letting it grow indefinitely, keeping computation cost constant regardless of how many views have been generated.
Target Context Routing (TCR) rotates which views are grouped together during the generation process: views are shuffled together early (high-noise steps) to agree on overall body structure, then locked into fixed neighboring groups later (low-noise steps) to polish fine details.
3D-aware skeleton conditioning: instead of dense depth maps (hard to estimate reliably), the system uses accurate but sparse 3D body skeletons rendered with proper depth ordering, resolving front/back ambiguity that flat 2D skeletons cannot.
On DNA-Rendering and DyMVHumans test datasets, 4DAnyone outperformed baseline methods (MV-Performer, TrajectoryCrafter, and a fine-tuned ReCamMaster) in generated video quality, cross-view consistency, and final 4D reconstruction quality, and generalized well to casual in-the-wild videos.

Figure 1. Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results.One continuous 3D scene: a smartphone at the lower left films a bass-playing subject, a ring of twenty-four generated camera views surrounds the stage, and the reconstructed subject stands at the center on a turntable, rendered from a free viewpoint.

Table 1. Training dataset statistics.
Dataset	Videos	Cameras	Actors	Resolution	Type
MVGameHuman	38k	24	318	2560×1440	Multi-view
SynCamVideo	34k	10	66	1280×1280	Multi-view
DNA-Rendering	51k	48	548	2048×2048	Multi-view
TedTalk	42k	1	413	2160×1620	Monocular
Pexels	20k	1	1,411	3840×2160	Monocular

Figure 2. Overview of 4DAnyone. Given a source video, an HMR model (GVHMR (33)) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for v target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS (39).Pipeline diagram. A source video is converted into skeleton-conditioned target-view tokens. Reference Context Packing supplies compact reference tokens, Target Context Routing exchanges information among target-view groups, and the generated multiview videos are reconstructed as a 4D Gaussian Splatting model.

Table 2. Quantitative comparison on DNA-Rendering and DyMVHumans. We evaluate 4DGS reconstruction, generated video consistency, and generated video reconstruction. Best results are in bold. † denotes our fine-tuned version.
	Method	DNA-Rendering	DyMVHumans
	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
Gen. Video Consistency	MV-Performer (59)	21.25	0.799	0.222	19.98	0.797	0.182
TrajectoryCrafter (52)	13.56	0.641	0.331	15.19	0.769	0.204
ReCamMaster† (2)	21.47	0.806	0.210	21.94	0.833	0.142
Ours	24.33	0.862	0.163	24.48	0.862	0.109
4DGS Reconstruction	MV-Performer (59)	20.38	0.826	0.191	18.69	0.787	0.158
TrajectoryCrafter (52)	14.81	0.718	0.331	15.11	0.748	0.217
ReCamMaster† (2)	20.55	0.807	0.214	19.86	0.795	0.159
Ours	24.15	0.863	0.159	23.28	0.846	0.117
Gen. Video Reconstruction	MV-Performer (59)	19.33	0.803	0.204	14.36	0.731	0.230
TrajectoryCrafter (52)	13.68	0.698	0.358	14.11	0.725	0.247
ReCamMaster† (2)	20.74	0.809	0.204	19.18	0.778	0.168
Ours	23.69	0.850	0.165	21.03	0.808	0.143

Figure 3. Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions v target views into m four-view groups, cyclically regroups them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement.Three-stage inference diagram. Initial views become a fixed multiscale reference context; target views are cyclically regrouped at high noise to share global structure and arranged into fixed adjacent groups at low noise to refine local details.

Table 3. Ablation study. Following the Gen. Video Consistency setting, we evaluate the consistency among generated videos.
Configuration	PSNR↑	SSIM↑	LPIPS↓
w/o TCR & RCP	21.09	0.766	0.216
w/o RCP	22.03	0.780	0.203
w/o TCR	22.21	0.788	0.196
Full (Random)	22.20	0.788	0.197
Full (Strided)	22.06	0.786	0.198
Full (Sliding)	22.63	0.796	0.191

Figure 4. Qualitative comparison with baselines. We show target-view generated videos (Gen.) and their corresponding 4DGS renderings (Rend.) across diverse human-centric videos. 4DAnyone produces geometrically accurate and visually detailed results across viewpoints, while the baselines suffer from inaccurate camera control (our fine-tuned ReCamMaster†) or geometric distortions (MV-Performer). See the project page for dynamic results.A grid compares generated target views and reconstructed 4D renderings from several methods on diverse human videos. The proposed method preserves body geometry, clothing appearance, and camera viewpoint more consistently than ReCamMaster and MV-Performer.

Table 4. Training data sampling. Source cameras are uniformly sampled from the listed options.
Dataset	No. Tgt Cam	No. Src Cam	No. Frame	Weight
MVGameHuman	6	1 / 4 / 8	41	0.4
MVGameHuman	4	1 / 4 / 8	61	0.4
MVGameHuman	1	1 / 4 / 8	121	0.2
SynCamVideo	4	1 / 4	61	0.8
SynCamVideo	1	1 / 4 / 8	81	0.2
DNA-Rendering	6	1 / 4 / 8	41	0.4
DNA-Rendering	4	1 / 4 / 8	61	0.4
DNA-Rendering	1	1 / 4 / 8	121	0.2
Pexels	1	1	121	1.0
TedTalk	1	1	121	1.0

Figure 5. Qualitative ablation results. Left: 3D-aware skeleton conditioning resolves the inherent ambiguity of 2D skeletons, guiding the model to generate geometrically correct content. Right: RCP and TCR maintain an effective cross-view context, leading to consistent appearance across generated views.Side-by-side ablation grids compare outputs with and without 3D-aware skeleton conditioning, Reference Context Packing, and Target Context Routing. The complete model has more accurate body geometry and more consistent appearance across views.

Table 5. Stage-specific training settings. “Indep. Src Prob” denotes the probability of independently sampling the source frame range. Skeleton: B=body, H=hands, F=feet, Fi=fingers.
Stage	Dataset	Bg. Removal	Indep. Src Prob	Skeleton
1	DNA-Rendering	✓	0.2	B+H+F+Fi
2	+ MVGameHuman, SynCam.	–	0.0	B+H+F+Fi
3	+ Pexels, TedTalk	–	0.0	B+H+F

Figure 6. Challenging cases. 4DAnyone robustly handles back-view source videos (left), complex subject appearance (right), and complex human motions.Two challenging examples show source frames and generated viewpoints for a person initially seen from the back and a person with complex clothing and appearance.

Table 6. Multi-GPU inference configurations. Configurations are shown for different camera setups.
No. Layers	No. Cam / Layer	No. GPUs	No. Tgt Cam / GPU
1	16	4	4
2	16	8	4
3	16	8	6

Figure 7. Robust generalization to diverse in-the-wild human videos. For each example, we show the source video (left), generated target-view videos (middle, 4 of 16 views), and a 4DGS novel-view rendering (right). See the project page for dynamic results.Sixteen in-the-wild examples are arranged in two columns. Each example shows a source human video frame, four generated target viewpoints, and a novel-view rendering of the reconstructed 4D Gaussian avatar.

Table 7. TCR switching-time sweep. Gen. Video Consistency when varying the number of sliding denoising steps.
ts/T	Sliding steps	PSNR↑	SSIM↑	LPIPS↓
1.00	0	22.2079	0.7880	0.1964
0.75	5	22.3030	0.7898	0.1947
0.50	10	22.4575	0.7925	0.1933
0.25	15	22.6093	0.7955	0.1912
0.20	16	22.6294	0.7963	0.1906
0.15	17	22.6023	0.7961	0.1908
0.10	18	22.6217	0.7967	0.1905
0.05	19	22.6367	0.7972	0.1903
0.00	20	22.6414	0.7971	0.1903

Figure 8. Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation.Eight rows of synthetic human scenes, each viewed simultaneously from four evenly spaced virtual cameras. The samples vary in actor, clothing, pose, lighting, environment, and background.

Why it matters

This approach lowers the barrier to creating realistic, freely viewable 3D human avatars from something as simple as a phone video, instead of requiring expensive multi-camera studio rigs. For anyone building AR/VR content, virtual production, or digital humans, it points toward a practical generation-then-reconstruct pipeline that avoids specialized hardware.

Figure 9. Single image to 4D avatar. Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS.A left-to-right pipeline turns one portrait into a pose-driven source video, generates synchronized target-view videos with 4DAnyone, and reconstructs an animatable 4D Gaussian avatar.

Terms in this paper

4D Gaussian Splatting (4DGS) · A technique that represents a moving 3D scene as a cloud of small 'Gaussian' blobs that can be rendered in real time from any viewpoint and any moment in time.
DiT (Diffusion Transformer) · A neural network architecture that generates images/videos by gradually removing noise, using a Transformer instead of the older U-Net design.
attention context · The set of information tokens a model can directly compare against each other in one pass; larger context lets more views be checked against each other at once, but costs more memory and compute.
HMR (Human Mesh Recovery) · AI methods that estimate a 3D body shape and pose (skeleton/mesh) from ordinary video.
z-buffer / depth-buffered rendering · A rendering technique that keeps track of which surface is nearest to the camera at each pixel, so closer body parts correctly hide farther ones instead of overlapping ambiguously.

Figure 10. Failure cases. Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box).Two failure examples. Red boxes highlight inconsistent large flowing fabric in one reconstruction and incorrect feet in generated views caused by a source-pose estimation error marked with a green circle.

Original abstract (English)

We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as O(N), weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with O(1) reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.

Authors · Yudong Jin

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Yudong Jin et al., arXiv:2608.20335, arxiv-nonexclusive