K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space

arXiv:2608.013972026-08-01

让机器人用生成动作的同一个策略来预测未来,比用单独的预测模块效果更好

SG-WAM是一种机器人策略模型,它在生成下一步动作的同时,用同一套策略表示来预测场景将如何变化,而不是依赖单独的未来预测模块。它用策略自身的缓慢更新副本(EMA目标)作为预测目标,并用一个冻结的3D几何模型给视觉特征注入空间结构信息。该方法在LIBERO仿真基准上平均成功率达98.5%,在更具挑战性的LIBERO-Plus上达73.0%,在真实机器人实验中也优于对比基线。

METAL MEDIA 解读图

SG-WAM架构:用策略自身的表示预测未来

证据状态已报告实测结果

  1. 观测+语言+动态令牌输入当前多视角摄像头图像、任务指令与8个可学习动态令牌一起输入共享的VLM主干(Qwen3.5-0.8B)
  2. 几何监督(VGGT教师)冻结的3D模型VGGT通过余弦相似度损失,对主视角视觉令牌进行塑形,注入空间结构信息
  3. 自引导世界预测器(SGWP)接收当前动态令牌状态和机器人即将执行的动作序列,预测动态令牌未来的演变
  4. EMA目标路径同一策略的缓慢更新副本处理未来观测,生成预测目标,且梯度被阻断
  5. 流匹配动作生成器以包含动态令牌的完整策略上下文为条件生成连续机器人动作;推理时教师模型、预测器和EMA路径全部被移除
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 现有的世界动作模型(WAM)要么直接预测未来图像/视频,导致模型把精力浪费在纹理、光照等与操作无关的视觉细节上;要么在一个单独的潜空间里预测特征,但这个空间未必与真正用于生成动作的表示对齐。
  2. SG-WAM在一个视觉语言模型(VLM)中插入8个可学习的动态令牌(dynamics tokens),并训练一个自引导世界预测器(SGWP)来预测这些令牌在机器人即将执行的动作作用下将如何变化。预测目标由同一策略的EMA(指数滑动平均)副本在看到未来观测后生成,使预测值和目标值来自同一表示体系。
  3. 与此同时,一个冻结的3D基础模型VGGT通过余弦相似度损失,对策略的主视角视觉令牌进行几何塑形,使动态令牌因此也携带场景的空间位置信息。
  4. 几何塑形、未来预测和负责生成实际动作的流匹配(flow matching)动作生成器在同一阶段联合训练;而在推理(实际部署)时,几何教师、SGWP预测分支和EMA目标路径全部被移除,只留下轻量级的部署策略。
  5. SG-WAM在LIBERO四个标准任务套件上平均成功率达到98.5%,在引入相机视角、光照、背景等扰动的零样本迁移基准LIBERO-Plus上以73.0%的总成功率领先,并在使用UR5e机械臂的三项真实任务(拾取放置、毛巾折叠、工具箱整理)中,在熟悉环境和背景/光照/新物体等未见条件下均优于VPP和VLA-JEPA基线。
Figure 1: Conceptual comparison of existing WAMs and SG-WAM. Whereas explicit and auxiliary latent targets may introduce perceptual burden or target–policy mismatch, SG-WAM learns intervening-action-conditioned dynamics in geometry-structured, policy-derived representations.
Figure 1: Conceptual comparison of existing WAMs and SG-WAM. Whereas explicit and auxiliary latent targets may introduce perceptual burden or target–policy mismatch, SG-WAM learns intervening-action-conditioned dynamics in geometry-structured, policy-derived representations.
Table 1: Simulation results on LIBERO. Success rates are reported for the four standard LIBERO suites. Bold indicates the best result, and underlining indicates the second-best result.
MethodParamsEmbodied PT.SpatialObjectGoalLongAvg.
OpenVLA-OFT [21]7B✓97.698.497.994.597.1
π0 [6]3.3B✓98.096.894.488.494.4
π0-FAST [31]3.3B✓96.496.888.660.285.5
π0.5 [5]3.3B✓98.898.298.092.496.9
GR00T N1.6 [4]3B✓97.798.597.594.497.0
Spatial Forcing [24]7B✓99.499.698.896.098.5
WorldVLA [10]7B✗87.696.283.460.081.8
LAPA [44]7B✓55.458.874.673.865.7
RynnVLA-002 [9]7B✗99.099.896.494.497.4
Mantis [42]5.8B✓98.899.294.494.296.7
UniVLA [8]7B✓96.596.895.692.095.2
Fast-WAM [45]6B✗98.2100.097.095.297.6
VLA-JEPA [34]2B✗94.899.695.894.096.1
SG-WAM0.9B✗99.499.898.696.298.5
Figure 2: Overview of the SG-WAM framework. The VLM jointly contextualizes multi-view observations, language, and learnable dynamics tokens for action generation. During training, a frozen VGGT teacher shapes main-view image tokens, while SGWP predicts future dynamics-token states conditioned on intervening actions and aligns them with an EMA policy target. All auxiliary branches are removed at inference.
Figure 2: Overview of the SG-WAM framework. The VLM jointly contextualizes multi-view observations, language, and learnable dynamics tokens for action generation. During training, a frozen VGGT teacher shapes main-view image tokens, while SGWP predicts future dynamics-token states conditioned on intervening actions and aligns them with an EMA policy target. All auxiliary branches are removed at inference.
Table 2: Zero-shot transfer results on LIBERO-Plus. Success rates are reported under different perturbation settings. Bold indicates the best result, and underlining indicates the second-best result.
MethodParamsCameraRobotLanguageLightBackgroundNoiseLayoutOverall
WorldVLA [10]7B0.127.941.643.717.110.938.025.0
Spatial Forcing [24]7B20.113.440.929.133.425.739.329.1
Mantis [42]5.8B15.741.845.945.128.939.262.539.8
UniVLA [8]7B4.350.371.859.180.025.334.341.5
Fast-WAM [45]6B16.444.568.978.253.737.760.750.0
π0 [6]3.3B13.86.058.885.081.479.068.953.6
VLA-JEPA [34]2B40.355.772.988.270.538.274.662.9
OpenVLA-OFT [21]7B56.431.979.588.793.375.874.269.6
SG-WAM0.9B58.648.981.489.886.180.774.273.0
Figure 3: Left: Visualization of the real-world platform. We use a UR5e robot arm as the manipulation platform, the Kinect Azure camera as the main camera and the RealSense D405 as the gripper camera. Right: Visualization of 3 real-world tasks. 1) Top: Pick and Place. 2) Middle: Towel Folding. 3) Bottom: Toolbox Organization.
Figure 3: Left: Visualization of the real-world platform. We use a UR5e robot arm as the manipulation platform, the Kinect Azure camera as the main camera and the RealSense D405 as the gripper camera. Right: Visualization of 3 real-world tasks. 1) Top: Pick and Place. 2) Middle: Towel Folding. 3) Bottom: Toolbox Organization.
Table 3: Success rates under different visual perturbations. Bold indicates the best result.
ModelPick and PlaceTowel FoldingToolbox Organization
IDBackgroundLight ChangeNovel ObjectIDBackgroundLight ChangeNovel ObjectID
VLA-JEPA35%20%25%20%20%10%10%15%20%
VPP30%15%10%10%35%15%15%10%30%
SG-WAM75%55%60%40%45%25%35%25%50%
Figure 4: Representative middle-layer attention maps from dynamics tokens to main-view image tokens. The task is Pick up the black bowl between the plate and the ramekin and place it on the plate. With geometric supervision, the selected tokens attend more consistently to the robot–object interaction regions.
Figure 4: Representative middle-layer attention maps from dynamics tokens to main-view image tokens. The task is Pick up the black bowl between the plate and the ramekin and place it on the plate. With geometric supervision, the selected tokens attend more consistently to the robot–object interaction regions.
Table 4: Ablation study of geometric supervision and self-guided world modeling on LIBERO. Geo. denotes geometric supervision, and WM. denotes self-guided world modeling. Success rates (%) are reported. Bold indicates the best result.
Geo.WM.SpatialObjectGoalLongAvg.
✗✗97.097.695.691.095.3
✓✗98.297.898.092.296.6
✗✓97.899.898.294.497.6
✓✓99.499.898.696.298.5
Figure 5: Overview of the Self-Guided World Predictor (SGWP). The projected dynamics-token states are first contextualized through self-attention and then attend to the encoded intervening actions through cross-attention to predict future latent states.
Figure 5: Overview of the Self-Guided World Predictor (SGWP). The projected dynamics-token states are first contextualized through self-attention and then attend to the encoded intervening actions through cross-attention to predict future latent states.
Table 5: Ablation study on the number of learnable dynamics tokens on LIBERO. Success rates (%) are reported. Bold indicates the best result.
Number of TokenSpatialObjectGoalLongAvg.
197.298.898.290.296.1
499.098.497.495.297.5
899.499.898.696.298.5
1699.499.497.692.497.2
Figure 6: Complete visualization of the Pick and Place task.
Figure 6: Complete visualization of the Pick and Place task.
Table 6: Subtasks success rates of Towel Folding and Toolbox Organization tasks in ID settings. Bold indicates the best result.
ModelTowel FoldingToolbox Organization
First FoldingSecond FoldingPick the ScrewdriverPick the First GearPick the Second GearClose the Toolbox
VLA-JEPA50%20%50%40%20%20%
VPP60%35%70%50%40%30%
SG-WAM75%45%80%60%50%50%
Figure 7: Complete visualization of the Towel Folding task.
Figure 7: Complete visualization of the Towel Folding task.
Table 7: Ablation of the action information provided to the Self-Guided World Predictor on LIBERO. For the null-action variant, the ground-truth intervening action sequence is replaced with an all-zero sequence before being processed by the action encoder. The action-conditioning architecture, temporal positional embeddings, and all other model components and training settings remain unchanged. Success rates (%) are reported.
VariantSpatialObjectGoalLongAvg.
Null-Action Sequence98.499.298.094.697.6
SG-WAM99.499.898.696.298.5
Figure 8: Complete visualization of the Toolbox Organization task.
Figure 8: Complete visualization of the Toolbox Organization task.

研究结果

  • 在LIBERO四个标准任务套件上,SG-WAM平均成功率达到98.5%,与使用更大模型或大规模具身预训练的最强基线持平。
  • 在引入相机视角、光照、背景、语言、场景布局等扰动的零样本迁移测试LIBERO-Plus上,SG-WAM以73.0%的总成功率排名第一,尤其在相机、语言、光照和布局扰动下表现最佳。
  • 在使用UR5e机械臂进行的拾取放置、毛巾折叠、工具箱整理三项真实任务中,SG-WAM在熟悉环境以及背景变化、光照变化、新物体等条件下,均优于VPP(显式视频预测)和VLA-JEPA(隐式潜空间预测)基线。
  • 同时移除几何监督和自引导世界建模后,LIBERO平均成功率降至95.3%;两者结合时达到最佳的98.5%。仅移除世界建模会损失1.9个百分点,仅移除几何监督损失0.9个百分点,在长时序任务LIBERO-Long上差距最大(从96.2%降到92.2%)。
  • 将可学习动态令牌数量从1个增加到8个,平均成功率从96.1%提升到98.5%,但继续增加到16个时反而降到97.2%。把真实的中间动作序列替换为全零序列后,平均成功率从98.5%降到97.6%,说明真实动作信息对预测提供了可测量但有限的帮助。
Figure 10: Visualization of the attention weight matrix of all latent dynamics tokens attending to main-view image tokens. w/ Geo. and w/o Geo. denote with geometric supervision and without geometric supervision, respectively.
Figure 10: Visualization of the attention weight matrix of all latent dynamics tokens attending to main-view image tokens. w/ Geo. and w/o Geo. denote with geometric supervision and without geometric supervision, respectively.

可应用场景

  • 希望用相对较小的模型同时获得强操作性能和更好的视觉分布外鲁棒性的机器人学习系统设计。
  • 希望在不生成昂贵未来图像或视频的前提下,让策略学习动作如何改变环境的机器人策略研究。
  • 希望减少多步骤长时序操作任务(如按顺序整理多个物体)中错误累积问题的研究。
Figure 11: Complete visualization of the Pick and Place task under the background shift.
Figure 11: Complete visualization of the Pick and Place task under the background shift.

局限与待验证事项

  • 真实实验仅限于UR5e机械臂、特定相机配置和三项任务(拾取放置、毛巾折叠、工具箱整理),尚未验证在其他机器人形态或更广泛任务上的泛化能力。
  • 结果基于一个相对较小的0.9B参数模型,且未经大规模具身预训练;作者明确将扩展到更大策略骨干网络和跨具身数据集列为未来工作。
  • 动态令牌数量的影响呈非单调趋势(增加到16个时性能反而下降),这一最优令牌数是否能直接迁移到其他任务或模型规模尚不确定。
  • 真实环境的分布外评估仅涵盖背景变化、光照变化、新物体三种具体扰动,对更极端或其他类型环境变化的鲁棒性尚未验证。
Figure 12: Complete visualization of the Pick and Place task under the light change.
Figure 12: Complete visualization of the Pick and Place task under the light change.

为什么重要

这项工作说明,机器人策略可以通过预测自身内部状态的未来变化(而非重建像素或依赖不匹配的辅助潜空间)来同时获得更强的操作性能和分布外鲁棒性。而且这一效果是在一个相对较小的0.9B参数模型上、且未经大规模具身预训练的情况下取得的,这对计算资源有限的团队有实际意义。

本文术语

  • 世界动作模型(World Action Model, WAM) · 在生成机器人动作的同时,联合学习预测环境如何随动作变化的模型
  • 动态令牌(dynamics tokens) · 加入模型中的可学习向量,用来编码场景在动作作用下的变化
  • EMA(指数滑动平均)目标 · 模型自身的一个缓慢更新副本,用于生成稳定的预测目标,且不参与梯度反传
  • 流匹配(flow matching) · 一种生成方法,通过逐步把随机噪声转换为目标输出(这里是连续的机器人动作)来生成结果
  • VGGT · 一个冻结的3D几何基础模型,训练时用于给策略的视觉特征注入空间结构信息

无法转载的图表

  • Figure 9: Visualization of 3 OOD settings. 1) Left: Background Shift. 2) Middle: Light Change. 3) Right: Novel Object.
在原文中查看图表 →

论文原文摘要（英文）

World Action Models (WAMs) couple action generation with prediction of future states. Their effectiveness depends on whether future dynamics are modeled in a space that is both aligned with action generation and sufficiently geometry-aware to capture where and how actions change the scene. Existing WAMs typically satisfy only part of this requirement, relying on either perceptually heavy observation-space targets or auxiliary latent spaces that are not jointly structured for action relevance and

作者 · Ruiteng Zhao

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Ruiteng Zhao et al., arXiv:2608.01397, arxiv-nonexclusive