SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space

What they did

1. Existing World Action Models (WAMs) either predict future images/video directly, which forces the model to waste capacity on irrelevant visual details like texture and lighting, or predict features in a separate latent space that may not match what the action-generation part of the model actually needs.
2. SG-WAM inserts 8 learnable 'dynamics tokens' into a vision-language model (VLM) and trains a Self-Guided World Predictor (SGWP) to forecast how these tokens will change under the robot's upcoming actions. The prediction target is generated by an EMA (exponential moving average) copy of the same policy looking at the future observation, so both the prediction and its target come from the same representation family.
3. In parallel, a frozen 3D foundation model called VGGT is used to shape the policy's main-view visual tokens with geometric structure via a cosine-similarity loss, so the dynamics tokens inherit spatial awareness of where things are in the scene.
4. All components (geometric shaping, future prediction, and a flow-matching action generator that produces the actual robot actions) are trained jointly in one stage; at inference time the geometry teacher, the SGWP branch, and the EMA target pathway are all removed, leaving only the lightweight deployed policy.
5. SG-WAM achieved 98.5% average success across four LIBERO suites, 73.0% on the more challenging zero-shot LIBERO-Plus benchmark (best among compared methods), and outperformed VPP and VLA-JEPA baselines on three real-world UR5e robot tasks (pick-and-place, towel folding, toolbox organization) under both familiar and novel conditions.

Findings

• On the four standard LIBERO suites, SG-WAM reached 98.5% average success, matching the strongest baseline despite using a much smaller model without large-scale embodied pretraining.
• On LIBERO-Plus, a zero-shot transfer test with shifts in camera viewpoint, lighting, background, language, and layout, SG-WAM achieved the highest overall success rate of 73.0%, notably best under camera, language, illumination, and layout shifts.
• In real-world tests on pick-and-place, towel folding, and toolbox organization with a UR5e arm, SG-WAM outperformed the VPP (explicit video-prediction) and VLA-JEPA (implicit latent-prediction) baselines both in-distribution and under background shift, light change, and novel-object conditions.
• Removing both geometric supervision and self-guided world modeling dropped average LIBERO success to 95.3%; combining both gave the best 98.5%. Removing world modeling alone cost 1.9 points, removing geometric supervision alone cost 0.9 points, with the largest gap on the long-horizon LIBERO-Long suite (96.2% to 92.2%).
• Increasing the number of dynamics tokens from 1 to 8 raised average success from 96.1% to 98.5%, but increasing further to 16 tokens reduced it to 97.2%. Replacing the real intervening actions with an all-zero sequence lowered average success from 98.5% to 97.6%, showing the actual action information provides a measurable, though modest, benefit.

Where it can be used

• Designing robot learning pipelines that want both strong in-distribution manipulation performance and better robustness to unseen visual conditions, using a relatively small model.
• Building robot policies that model how actions change the environment without generating expensive future images or video.
• Reducing compounding failures in long, multi-step manipulation tasks such as sequentially organizing multiple objects.

Limits and open work

• Real-world experiments are limited to a UR5e arm, a specific camera setup, and three tasks (pick-and-place, towel folding, toolbox organization); generalization to other robot embodiments or a much wider range of tasks is untested.
• Results come from a comparatively small 0.9B-parameter model without large-scale embodied pretraining; the authors explicitly leave scaling to larger backbones and cross-embodiment datasets as future work.
• The number of dynamics tokens shows a non-monotonic effect (performance drops at 16 tokens), so the optimal token count found here may not transfer directly to other tasks or model scales.
• Real-world out-of-distribution evaluation covers only three specific shift types (background, lighting, novel object), so robustness to more extreme or different kinds of environmental changes remains unverified.

Why it matters

This work suggests that robot policies can gain robustness and manipulation performance by predicting their own future internal states rather than reconstructing pixels or relying on a mismatched auxiliary latent space. It does this with a relatively small 0.9B-parameter model and no large-scale robot pretraining, which matters for teams with limited compute.

Terms in this paper

• World Action Model (WAM) · A robot model that jointly generates actions and predicts how the environment will change as a result
• dynamics tokens · Learnable vectors added to the model to encode how the scene changes as a result of robot actions
• EMA (exponential moving average) target · A slowly-updated copy of the model used to produce stable prediction targets without back-propagating gradients into it
• flow matching · A generation technique that transforms random noise into a target output (here, continuous robot actions) step by step
• VGGT · A frozen 3D geometry foundation model used during training to inject spatial structure into the policy's visual features

Figures we cannot republish

• Figure 9: Visualization of 3 OOD settings. 1) Left: Background Shift. 2) Middle: Light Change. 3) Right: Novel Object.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctlyarXiv:2608.194082026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21

Latest from METAL MEDIA

• TUIDE debut with 'SUN KISS,' an Iceland-shot K-pop road movie
• NCT 127 drop BLINGY, their first full album in two years