K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

4DAnyone: Create Anyone in 4D from a Casual Monocular Video

arXiv:2608.203352026-08-19

用手机随手拍的一段视频,生成可以任意角度环绕观看的4D人物模型

4DAnyone只需一段普通的单目人物视频,就能生成该人物在数十个虚拟机位下保持一致的视频画面,再把这些画面重建成4D高斯泼溅(4DGS)模型,从而在任意时刻、任意视角自由渲染这个人。现有的相机可控视频生成模型在少量新视角上表现尚可,但一旦扩展到3D重建所需的数十个视角时就会出现明显的不一致,作者提出了两项新技术——Reference Context Packing和Target Context Routing,并配合自建的合成数据集MVGameHuman来解决这一问题。在两个基准数据集上,该方法优于此前的方法。

METAL MEDIA 解读图

用手机随手拍的一段视频,生成可以任意角度环绕观看的4D人物模型

  1. 01问题所在:现有相机可控视频生成模型在生成少量新视角视频时看起来还不错,但一旦扩展到4DGS重建所需的十几个甚至更多视角,由于模型单次处理能显式关联的信息范围有限,各视角之间会出现结构漂移和外观不一致。
  2. 02Reference Context Packing(RCP)不再让已生成的参考视角视频无限累积,而是把它们压缩进一个长度固定、分辨率混合的上下文中,使参考信息的计算开销不随视角数量增加而增长。
  3. 03Target Context Routing(TCR)在去噪过程中轮换视角分组:在噪声较大的早期阶段打乱分组、让不同组交换信息以统一整体身体结构,在噪声较小的后期阶段则固定相邻视角分组,专注细化局部细节。
  4. 04三维感知骨骼条件:相比难以可靠估计的密集深度图,该方法采用精度更高但更稀疏的三维骨骼信息,并通过深度排序渲染(z缓冲)解决了普通二维骨骼图无法区分前后遮挡的问题。
  5. 05在DNA-Rendering和DyMVHumans两个测试数据集上,4DAnyone在生成视频质量、跨视角一致性以及最终4D重建效果上均优于MV-Performer、TrajectoryCrafter以及经过相同方式微调的ReCamMaster等对比方法,并且在真实场景随手拍摄的视频上也表现出良好的泛化能力。
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 问题所在:现有相机可控视频生成模型在生成少量新视角视频时看起来还不错,但一旦扩展到4DGS重建所需的十几个甚至更多视角,由于模型单次处理能显式关联的信息范围有限,各视角之间会出现结构漂移和外观不一致。
  2. Reference Context Packing(RCP)不再让已生成的参考视角视频无限累积,而是把它们压缩进一个长度固定、分辨率混合的上下文中,使参考信息的计算开销不随视角数量增加而增长。
  3. Target Context Routing(TCR)在去噪过程中轮换视角分组:在噪声较大的早期阶段打乱分组、让不同组交换信息以统一整体身体结构,在噪声较小的后期阶段则固定相邻视角分组,专注细化局部细节。
  4. 三维感知骨骼条件:相比难以可靠估计的密集深度图,该方法采用精度更高但更稀疏的三维骨骼信息,并通过深度排序渲染(z缓冲)解决了普通二维骨骼图无法区分前后遮挡的问题。
  5. 在DNA-Rendering和DyMVHumans两个测试数据集上,4DAnyone在生成视频质量、跨视角一致性以及最终4D重建效果上均优于MV-Performer、TrajectoryCrafter以及经过相同方式微调的ReCamMaster等对比方法,并且在真实场景随手拍摄的视频上也表现出良好的泛化能力。
Figure 1. Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results.One continuous 3D scene: a smartphone at the lower left films a bass-playing subject, a ring of twenty-four generated camera views surrounds the stage, and the reconstructed subject stands at the center on a turntable, rendered from a free viewpoint.
Figure 1. Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results.One continuous 3D scene: a smartphone at the lower left films a bass-playing subject, a ring of twenty-four generated camera views surrounds the stage, and the reconstructed subject stands at the center on a turntable, rendered from a free viewpoint.
Table 1. Training dataset statistics.
DatasetVideosCamerasActorsResolutionType
MVGameHuman38k243182560×1440Multi-view
SynCamVideo34k10661280×1280Multi-view
DNA-Rendering51k485482048×2048Multi-view
TedTalk42k14132160×1620Monocular
Pexels20k11,4113840×2160Monocular
Figure 2. Overview of 4DAnyone. Given a source video, an HMR model (GVHMR (33)) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for v target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS (39).Pipeline diagram. A source video is converted into skeleton-conditioned target-view tokens. Reference Context Packing supplies compact reference tokens, Target Context Routing exchanges information among target-view groups, and the generated multiview videos are reconstructed as a 4D Gaussian Splatting model.
Figure 2. Overview of 4DAnyone. Given a source video, an HMR model (GVHMR (33)) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for v target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS (39).Pipeline diagram. A source video is converted into skeleton-conditioned target-view tokens. Reference Context Packing supplies compact reference tokens, Target Context Routing exchanges information among target-view groups, and the generated multiview videos are reconstructed as a 4D Gaussian Splatting model.
Table 2. Quantitative comparison on DNA-Rendering and DyMVHumans. We evaluate 4DGS reconstruction, generated video consistency, and generated video reconstruction. Best results are in bold. † denotes our fine-tuned version.
MethodDNA-RenderingDyMVHumans
PSNR↑SSIM↑LPIPS↓PSNR↑SSIM↑LPIPS↓
Gen. Video ConsistencyMV-Performer (59)21.250.7990.22219.980.7970.182
TrajectoryCrafter (52)13.560.6410.33115.190.7690.204
ReCamMaster† (2)21.470.8060.21021.940.8330.142
Ours24.330.8620.16324.480.8620.109
4DGS ReconstructionMV-Performer (59)20.380.8260.19118.690.7870.158
TrajectoryCrafter (52)14.810.7180.33115.110.7480.217
ReCamMaster† (2)20.550.8070.21419.860.7950.159
Ours24.150.8630.15923.280.8460.117
Gen. Video ReconstructionMV-Performer (59)19.330.8030.20414.360.7310.230
TrajectoryCrafter (52)13.680.6980.35814.110.7250.247
ReCamMaster† (2)20.740.8090.20419.180.7780.168
Ours23.690.8500.16521.030.8080.143
Figure 3. Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions v target views into m four-view groups, cyclically regroups them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement.Three-stage inference diagram. Initial views become a fixed multiscale reference context; target views are cyclically regrouped at high noise to share global structure and arranged into fixed adjacent groups at low noise to refine local details.
Figure 3. Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions v target views into m four-view groups, cyclically regroups them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement.Three-stage inference diagram. Initial views become a fixed multiscale reference context; target views are cyclically regrouped at high noise to share global structure and arranged into fixed adjacent groups at low noise to refine local details.
Table 3. Ablation study. Following the Gen. Video Consistency setting, we evaluate the consistency among generated videos.
ConfigurationPSNR↑SSIM↑LPIPS↓
w/o TCR & RCP21.090.7660.216
w/o RCP22.030.7800.203
w/o TCR22.210.7880.196
Full (Random)22.200.7880.197
Full (Strided)22.060.7860.198
Full (Sliding)22.630.7960.191
Figure 4. Qualitative comparison with baselines. We show target-view generated videos (Gen.) and their corresponding 4DGS renderings (Rend.) across diverse human-centric videos. 4DAnyone produces geometrically accurate and visually detailed results across viewpoints, while the baselines suffer from inaccurate camera control (our fine-tuned ReCamMaster†) or geometric distortions (MV-Performer). See the project page for dynamic results.A grid compares generated target views and reconstructed 4D renderings from several methods on diverse human videos. The proposed method preserves body geometry, clothing appearance, and camera viewpoint more consistently than ReCamMaster and MV-Performer.
Figure 4. Qualitative comparison with baselines. We show target-view generated videos (Gen.) and their corresponding 4DGS renderings (Rend.) across diverse human-centric videos. 4DAnyone produces geometrically accurate and visually detailed results across viewpoints, while the baselines suffer from inaccurate camera control (our fine-tuned ReCamMaster†) or geometric distortions (MV-Performer). See the project page for dynamic results.A grid compares generated target views and reconstructed 4D renderings from several methods on diverse human videos. The proposed method preserves body geometry, clothing appearance, and camera viewpoint more consistently than ReCamMaster and MV-Performer.
Table 4. Training data sampling. Source cameras are uniformly sampled from the listed options.
DatasetNo. Tgt CamNo. Src CamNo. FrameWeight
MVGameHuman61 / 4 / 8410.4
MVGameHuman41 / 4 / 8610.4
MVGameHuman11 / 4 / 81210.2
SynCamVideo41 / 4610.8
SynCamVideo11 / 4 / 8810.2
DNA-Rendering61 / 4 / 8410.4
DNA-Rendering41 / 4 / 8610.4
DNA-Rendering11 / 4 / 81210.2
Pexels111211.0
TedTalk111211.0
Figure 5. Qualitative ablation results. Left: 3D-aware skeleton conditioning resolves the inherent ambiguity of 2D skeletons, guiding the model to generate geometrically correct content. Right: RCP and TCR maintain an effective cross-view context, leading to consistent appearance across generated views.Side-by-side ablation grids compare outputs with and without 3D-aware skeleton conditioning, Reference Context Packing, and Target Context Routing. The complete model has more accurate body geometry and more consistent appearance across views.
Figure 5. Qualitative ablation results. Left: 3D-aware skeleton conditioning resolves the inherent ambiguity of 2D skeletons, guiding the model to generate geometrically correct content. Right: RCP and TCR maintain an effective cross-view context, leading to consistent appearance across generated views.Side-by-side ablation grids compare outputs with and without 3D-aware skeleton conditioning, Reference Context Packing, and Target Context Routing. The complete model has more accurate body geometry and more consistent appearance across views.
Table 5. Stage-specific training settings. “Indep. Src Prob” denotes the probability of independently sampling the source frame range. Skeleton: B=body, H=hands, F=feet, Fi=fingers.
StageDatasetBg. RemovalIndep. Src ProbSkeleton
1DNA-Rendering✓0.2B+H+F+Fi
2+ MVGameHuman, SynCam.–0.0B+H+F+Fi
3+ Pexels, TedTalk–0.0B+H+F
Figure 6. Challenging cases. 4DAnyone robustly handles back-view source videos (left), complex subject appearance (right), and complex human motions.Two challenging examples show source frames and generated viewpoints for a person initially seen from the back and a person with complex clothing and appearance.
Figure 6. Challenging cases. 4DAnyone robustly handles back-view source videos (left), complex subject appearance (right), and complex human motions.Two challenging examples show source frames and generated viewpoints for a person initially seen from the back and a person with complex clothing and appearance.
Table 6. Multi-GPU inference configurations. Configurations are shown for different camera setups.
No. LayersNo. Cam / LayerNo. GPUsNo. Tgt Cam / GPU
11644
21684
31686
Figure 7. Robust generalization to diverse in-the-wild human videos. For each example, we show the source video (left), generated target-view videos (middle, 4 of 16 views), and a 4DGS novel-view rendering (right). See the project page for dynamic results.Sixteen in-the-wild examples are arranged in two columns. Each example shows a source human video frame, four generated target viewpoints, and a novel-view rendering of the reconstructed 4D Gaussian avatar.
Figure 7. Robust generalization to diverse in-the-wild human videos. For each example, we show the source video (left), generated target-view videos (middle, 4 of 16 views), and a 4DGS novel-view rendering (right). See the project page for dynamic results.Sixteen in-the-wild examples are arranged in two columns. Each example shows a source human video frame, four generated target viewpoints, and a novel-view rendering of the reconstructed 4D Gaussian avatar.
Table 7. TCR switching-time sweep. Gen. Video Consistency when varying the number of sliding denoising steps.
ts/TSliding stepsPSNR↑SSIM↑LPIPS↓
1.00022.20790.78800.1964
0.75522.30300.78980.1947
0.501022.45750.79250.1933
0.251522.60930.79550.1912
0.201622.62940.79630.1906
0.151722.60230.79610.1908
0.101822.62170.79670.1905
0.051922.63670.79720.1903
0.002022.64140.79710.1903
Figure 8. Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation.Eight rows of synthetic human scenes, each viewed simultaneously from four evenly spaced virtual cameras. The samples vary in actor, clothing, pose, lighting, environment, and background.
Figure 8. Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation.Eight rows of synthetic human scenes, each viewed simultaneously from four evenly spaced virtual cameras. The samples vary in actor, clothing, pose, lighting, environment, and background.

为什么重要

这项方法为无需昂贵多机位摄影棚、仅用一段手机视频就能生成可自由观看的3D人物数字化身提供了可行路径。对于从事AR/VR内容制作、虚拟拍摄或数字人开发的人来说,它展示了一条无需专用硬件即可实现的实用生成再重建流程。

Figure 9. Single image to 4D avatar. Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS.A left-to-right pipeline turns one portrait into a pose-driven source video, generates synchronized target-view videos with 4DAnyone, and reconstructs an animatable 4D Gaussian avatar.
Figure 9. Single image to 4D avatar. Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS.A left-to-right pipeline turns one portrait into a pose-driven source video, generates synchronized target-view videos with 4DAnyone, and reconstructs an animatable 4D Gaussian avatar.

本文术语

  • 4D高斯泼溅(4DGS) · 用大量微小的高斯团块来表示会随时间变化的三维场景,可实时从任意视角和任意时刻进行渲染的技术
  • DiT(扩散Transformer) · 一种通过逐步去除噪声来生成图像或视频的扩散模型架构,用Transformer结构替代了早期的U-Net设计
  • 注意力上下文(attention context) · 模型在一次计算中能够互相比对参考的信息范围,范围越大能同时核对的视角越多,但所需内存和算力也越高
  • 人体网格恢复(HMR) · 从普通视频中估计人体三维形状与姿态(骨骼或网格)的AI技术
  • z缓冲深度渲染 · 记录每个像素处哪个表面离相机最近,使距离较近的身体部位能正确遮挡较远部位的一种渲染方法
Figure 10. Failure cases. Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box).Two failure examples. Red boxes highlight inconsistent large flowing fabric in one reconstruction and incorrect feet in generated views caused by a source-pose estimation error marked with a green circle.
Figure 10. Failure cases. Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box).Two failure examples. Red boxes highlight inconsistent large flowing fabric in one reconstruction and incorrect feet in generated views caused by a source-pose estimation error marked with a green circle.

论文原文摘要（英文）

We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as O(N), weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with O(1) reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.

作者 · Yudong Jin

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Yudong Jin et al., arXiv:2608.20335, arxiv-nonexclusive