K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

WorldClaw: Agentic 3D Open-World Generation at Scale

arXiv:2608.052482026-08-07

多个AI智能体协同合作,仅凭一句文字描述就能构建出可自由行走的整片3D开放世界

WorldClaw是一个智能体系统,输入一句开放式文字提示,就能生成包含地形、建筑、树木、船只等元素的大规模3D世界。规划智能体先把提示转化为包含区域、地形、物体的结构化设计方案,随后构建全局地形基础,再只针对需要细节的区域生成并放置具体物体。基于渲染结果进行检查和修改的智能体会反复迭代,让场景更加自然连贯。

METAL MEDIA 解读图

WorldClaw的三阶段全局到局部生成流程

证据状态实测结果与计划中的工作并存

  1. 1. 意图分析与规划将简短的用户提示转化为包含区域、地形、物体、材质与空间关系的结构化场景说明
  2. 2. 全局地形生成根据语义布局图和地形参数构建高度场,并撒布可复用资产与材质,形成连贯的整体地形
  3. 3. 区域物体生成与放置对需要细节的区域渲染地形图像,用图像编辑插入物体,再通过图生3D重建为独立网格并放置到地形上
  4. 4. 基于渲染的精修循环智能体检查渲染画面,反复修正地形接缝、材质比例以及物体姿态与地面接触问题
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 意图分析智能体与场景规划智能体先将用户简短的提示转化为结构化场景说明,涵盖区域划分、地形类型、物体类别、材质与空间关系。
  2. 根据该说明生成用颜色区分区域的2D语义布局图,并据此计算高度场(地形起伏),同时生成可复用的岩石、植被等3D资产及材质,构建出连贯的全局地形。
  3. 对于需要细节的区域,系统先把当地地形渲染成2D图像,再用图像编辑模型在其中添加建筑、船只等物体,然后将这些物体重建为独立的3D网格模型,并放置回地形上的正确位置。
  4. 基于渲染结果的精修智能体会检查画面,反复修正地形过渡不自然、材质比例失调,以及物体悬空、姿态或大小与地面不匹配等接触问题。
  5. WorldClaw在热带海盗要塞、峡谷部落聚落、沙漠战场、雪山峡谷基地等多种不同提示下,生成了具有连贯整体结构、可编辑独立3D资产的大规模场景。
Figure 1: Overview of WorldClaw. Given an open-ended text prompt, WorldClaw constructs an explicit, explorable, and editable 3D world through a three-stage global-to-regional pipeline. (1) Intent Analysis and Planning translates the prompt into a structured scene specification. (2) Global Terrain Generation establishes a region-aware terrain foundation with coherent geometry, appearance, and spatial semantics. (3) Regional Object Generation and Placement populates planned regions with editable 3D assets and refines their arrangements and interactions with terrain.
Figure 1: Overview of WorldClaw. Given an open-ended text prompt, WorldClaw constructs an explicit, explorable, and editable 3D world through a three-stage global-to-regional pipeline. (1) Intent Analysis and Planning translates the prompt into a structured scene specification. (2) Global Terrain Generation establishes a region-aware terrain foundation with coherent geometry, appearance, and spatial semantics. (3) Regional Object Generation and Placement populates planned regions with editable 3D assets and refines their arrangements and interactions with terrain.
Figure 2: Overview of global terrain generation and refinement.(a) Initial Height-Field Generation constructs composite terrain geometry from the semantic layout map and region-specific terrain parameters and assigns materials to the corresponding regions.(b) Global Terrain Asset Scattering instantiates terrain-associated assets according to regional semantics and local surface conditions.(c) Terrain Refinement renders, inspects, and locally edits the terrain to correct geometric transitions, material scales, asset distributions, and rendering artifacts.
Figure 2: Overview of global terrain generation and refinement.(a) Initial Height-Field Generation constructs composite terrain geometry from the semantic layout map and region-specific terrain parameters and assigns materials to the corresponding regions.(b) Global Terrain Asset Scattering instantiates terrain-associated assets according to regional semantics and local surface conditions.(c) Terrain Refinement renders, inspects, and locally edits the terrain to correct geometric transitions, material scales, asset distributions, and rendering artifacts.

研究结果

  • WorldClaw在热带海盗要塞、峡谷部落聚落、沙漠战场、雪山峡谷基地、中世纪村庄、雪地河畔村庄、沙漠冒险营地、日式风格岛屿小镇等多种提示下,实际生成了具有连贯全局空间组织和丰富区域内容的大规模场景。
  • 研究者以相同的中世纪村庄主题为条件,对WorldClaw与代表性的文本驱动3D场景生成方法进行了定性比较,但该比较未报告具体数值指标。
  • 作者观察到,目前的开源语言模型经常无法生成既可执行又符合用户需求的程序化地形与材质代码,开源图像生成模型也常常无法生成可用的布局图,或在物体图像生成与提取过程中难以保持物体外观和姿态。
  • 作者指出,目前要充分验证这一解耦流程,仍需依赖Claude Opus 4.8、GPT-Image-2、Hunyuan3D等能力较强的模型。
Figure 3: Scene refinement. (a) Object Refinement processes objects from a report queue, evaluates pose, mesh quality, and scale against semantic and regional context, applies targeted edits, and verifies the result through re-rendering. (b) Terrain Refinement examines support-surface quality and object-terrain collisions, applies local co-deformation to defects such as floating, and updates the report after re-rendering.
Figure 3: Scene refinement. (a) Object Refinement processes objects from a report queue, evaluates pose, mesh quality, and scale against semantic and regional context, applies targeted edits, and verifies the result through re-rendering. (b) Terrain Refinement examines support-surface quality and object-terrain collisions, applies local co-deformation to defects such as floating, and updates the report after re-rendering.
Figure 4: Tropical pirate stronghold. An island terrain organizes dense vegetation, settlements, docks, and ships into distinct coastal regions. The figure presents the global composition, regional close-ups, local walk views, and their corresponding instance, depth, and normal renderings.
Figure 4: Tropical pirate stronghold. An island terrain organizes dense vegetation, settlements, docks, and ships into distinct coastal regions. The figure presents the global composition, regional close-ups, local walk views, and their corresponding instance, depth, and normal renderings.

可应用场景

  • 仅凭文字快速起草游戏或VR环境中大规模地形与建筑布局的初稿
  • 为影视、动画的大规模外景预演准备可编辑的3D资产
  • 为机器人仿真或具身智能训练批量生成具有多样地形和物体布局的3D环境
  • 生成可导入Unreal Engine等游戏引擎、供进一步编辑和动画制作的带纹理网格素材
Figure 5: Canyon with tribal settlements. A continuous river connects the canyon, valley floor, vegetation, and settlement regions across substantial elevation changes. Global and regional views are complemented by local walk views and the corresponding instance, depth, and normal renderings.
Figure 5: Canyon with tribal settlements. A continuous river connects the canyon, valley floor, vegetation, and settlement regions across substantial elevation changes. Global and regional views are complemented by local walk views and the corresponding instance, depth, and normal renderings.
Figure 6: Thrilling battlefield in the desert. Layered rocky landforms surround open combat areas and populated compounds containing buildings, defensive structures, and vehicles. The figure shows the global and regional organization together with local walk views and their instance, depth, and normal renderings.
Figure 6: Thrilling battlefield in the desert. Layered rocky landforms surround open combat areas and populated compounds containing buildings, defensive structures, and vehicles. The figure shows the global and regional organization together with local walk views and their instance, depth, and normal renderings.

局限与待验证事项

  • 流程各阶段高度依赖外部大语言模型、图像生成模型和3D生成模型的能力,最终画面质量直接受限于这些底层模型的水平。
  • 大语言模型生成的Blender代码在尺度估计、数值参数或节点连接上经常出错,导致地形不协调、材质效果不准确或物体布局偏离用户意图,往往需要多轮渲染-检查-修正才能纠正。
  • 由于对物体单独生成重建,并对地形、资产及其接触关系进行多轮智能体式精修,该长流程带来了较高的推理延迟和计算成本,且随物体数量和精修迭代次数增加而进一步上升。
  • 对于较简单的场景,这一长流程相比一次性整体生成的方法可能显得效率偏低。
  • 当前物体重建仍主要依赖生成式3D模型,尚不能稳定恢复明确的部件层级结构、参数化结构、关节定义或交互逻辑,作者将其列为未来工作方向。
Figure 7: Snow-covered mountain valley with style of Command & Conquer: Red Alert. The enclosing mountain terrain contains multiple regions populated with futuristic facilities, communication structures, and vehicles. Global, regional, and local walk views are shown with the corresponding instance, depth, and normal renderings.
Figure 7: Snow-covered mountain valley with style of Command & Conquer: Red Alert. The enclosing mountain terrain contains multiple regions populated with futuristic facilities, communication structures, and vehicles. Global, regional, and local walk views are shown with the corresponding instance, depth, and normal renderings.
Figure 8: Qualitative comparison with the representative text-driven 3D scene generation methods. All methods are conditioned on prompts that share the same medieval-village theme and comparable scene requirements, with the wording adapted when necessary to each method’s input format.
Figure 8: Qualitative comparison with the representative text-driven 3D scene generation methods. All methods are conditioned on prompts that share the same medieval-village theme and comparable scene requirements, with the wording adapted when necessary to each method’s input format.

为什么重要

游戏、影视、VR和机器人仿真都需要可行走、可编辑的显式3D世界,但现有方法往往难以在保持整体一致性的同时填充丰富的局部细节。WorldClaw用先搭全局框架、再补局部细节的智能体式方法尝试解决这一矛盾,为把开放式文字直接转化为可用于实际生产的3D内容提供了一种参考流程。

Figure 9: Medieval-style village across diverse terrain. The scene combines snow-capped mountains, forested plains, waterways, and desert areas, with village buildings, windmills, vegetation, and animals distributed across the different regions. The figure shows the global layout, regional close-ups, local walk views, and their corresponding instance, depth, and normal renderings.
Figure 9: Medieval-style village across diverse terrain. The scene combines snow-capped mountains, forested plains, waterways, and desert areas, with village buildings, windmills, vegetation, and animals distributed across the different regions. The figure shows the global layout, regional close-ups, local walk views, and their corresponding instance, depth, and normal renderings.
Figure 10: Snow-covered riverside village. The village extends along both sides of a frozen river within a mountainous landscape. The figure shows the global layout, regional close-ups, and local walk views together with instance, depth, and normal renderings.
Figure 10: Snow-covered riverside village. The village extends along both sides of a frozen river within a mountainous landscape. The figure shows the global layout, regional close-ups, and local walk views together with instance, depth, and normal renderings.

本文术语

  • 智能体(Agentic) · 能够自主规划、调用工具、检查结果并进行修改的AI程序,而非一次性输出结果
  • 高度场(Height Field) · 用2D网格上的数值表示地形高低起伏的方式
  • 语义布局图 · 用不同颜色标记各区域地形类型或功能的2D地图
  • 图生3D(Image-to-3D) · 输入物体的2D图像,重建出该物体3D形状和外观的模型能力
  • 带纹理网格(Textured Mesh) · 表面贴有颜色和材质图像、可在常规3D工具中编辑的3D形状数据
Figure 11: Desert adventure camp surrounded by dragons. The scene combines layered desert terrain, settlements, watchtowers, and dragons distributed around the camp. Global, regional, and local walk views are shown with the corresponding instance, depth, and normal renderings.
Figure 11: Desert adventure camp surrounded by dragons. The scene combines layered desert terrain, settlements, watchtowers, and dragons distributed around the camp. Global, regional, and local walk views are shown with the corresponding instance, depth, and normal renderings.
Figure 12: Island with Japanese-style towns. Multiple settlements are distributed across an island containing coastlines, vegetation, hills, and water. The figure presents the global and regional organization as well as local walk views and their instance, depth, and normal renderings.
Figure 12: Island with Japanese-style towns. Multiple settlements are distributed across an island containing coastlines, vegetation, hills, and water. The figure presents the global and regional organization as well as local walk views and their instance, depth, and normal renderings.

论文原文摘要（英文）

Generating large-scale, freely explorable 3D worlds from open-ended text remains challenging because a system must jointly maintain global spatial coherence, rich local content, and explicit assets suitable for downstream editing and reuse. We present WorldClaw, a fully agentic, coarse-to-fine framework for open-world 3D scene generation. Planning agents translate a text prompt into a structured specification of regions, terrain, assets, materials, and spatial relations. WorldClaw then builds a globally coherent terrain foundation from semantic layouts, reusable assets, generative or procedural materials, and a region-aware height field. For detail-demanding regions, it generates terrain-conditioned compositions, reconstructs editable textured meshes, and recovers their placement on the terrain; render-based agents further refine terrain, objects, appearance, and contacts. Across diverse open-world prompts, WorldClaw produces large-scale scenes with coherent spatial organization, visually compelling local content, and editable instance-level assets while preserving a consistent global terrain structure.

作者 · Chunchao Guo, Jinpeng Li, Yang Li, Zilong Huang

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Chunchao Guo et al., arXiv:2608.05248, arxiv-nonexclusive