Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation
arXiv:2608.175122026-08-17
A navigation AI that just points at a spot on the camera image instead of forcing 3D commands
Existing robot navigation models struggled because they made vision-language AI learn awkward 3D actions like turning by exact angles, which clashed with how these models were originally trained on flat 2D images. TAMP-Nav instead lets the AI simply pick a 2D pixel on the camera view to point toward, then converts that point into a 3D location for the robot's low-level movement system to execute. This approach reached a 66.2% success rate on the R2R-CE benchmark using only 90,000 training trajectories, beating prior state-of-the-art systems in both accuracy and efficiency.
METAL MEDIA explanatory visual
A navigation AI that just points at a spot on the camera image instead of forcing 3D commands
01Vision-language models are pretrained on 2D images, so forcing them to output precise 3D turning angles or coordinates caused errors. TAMP-Nav instead has the model act like a 'pointer,' clicking a 2D pixel on the camera feed, which is then projected into 3D using depth data and handed to a SLAM (simultaneous localization and mapping) controller for actual movement.
02Instead of reasoning at every single step (slow) or never reasoning (inaccurate), the model learns to trigger Chain-of-Thought reasoning only at critical moments like crossroads or doorways. Ordinary stretches of the path are compressed into lightweight 'Space-Time Indicator' tokens that just record position and time, keeping memory efficient without losing key waypoints.
03Training uses a 'Two-Level GRPO' reinforcement learning method that combines rewards for whole completed trajectories with rewards for individual step-by-step decisions, covering things like collision avoidance, correct stopping, and whether reasoning was actually useful.
04The model achieved 66.2% success rate on R2R-CE and 65.7% on RxR-CE, state-of-the-art results, while averaging only 9 movement steps per trajectory versus about 30 for competing systems. It was also deployed on a real quadruped robot (Unitree Go2) trained purely in simulation and still worked in unseen indoor and outdoor environments.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Vision-language models are pretrained on 2D images, so forcing them to output precise 3D turning angles or coordinates caused errors. TAMP-Nav instead has the model act like a 'pointer,' clicking a 2D pixel on the camera feed, which is then projected into 3D using depth data and handed to a SLAM (simultaneous localization and mapping) controller for actual movement.
Instead of reasoning at every single step (slow) or never reasoning (inaccurate), the model learns to trigger Chain-of-Thought reasoning only at critical moments like crossroads or doorways. Ordinary stretches of the path are compressed into lightweight 'Space-Time Indicator' tokens that just record position and time, keeping memory efficient without losing key waypoints.
Training uses a 'Two-Level GRPO' reinforcement learning method that combines rewards for whole completed trajectories with rewards for individual step-by-step decisions, covering things like collision avoidance, correct stopping, and whether reasoning was actually useful.
The model achieved 66.2% success rate on R2R-CE and 65.7% on RxR-CE, state-of-the-art results, while averaging only 9 movement steps per trajectory versus about 30 for competing systems. It was also deployed on a real quadruped robot (Unitree Go2) trained purely in simulation and still worked in unseen indoor and outdoor environments.
Table 1: Comparison of different methods on R2R-CE and RxR-CE validation unseen splits.
You are a strict filter for navigation instructions.
- Pure action = a movement-only command without a target object/landmark/destination, e.g., ”go straight”, ”turn left”.
- Not pure action = includes a target or landmark, e.g., ”go straight to the bed”.
If there are 4 or more consecutive pure actions anywhere in the instruction, output DROP. Otherwise output KEEP. Consecutive means back-to-back pure actions with no non-pure-action instruction between them.
You are an expert at summarizing navigation trajectories. Given a task and image sequences, your job is to summarize the trajectory from image 0→k in ONE continuous paragraph. Explicitly state which part of the task has been completed. Do NOT mention future parts. Return ONLY a JSON object.
User Prompt Template
Task: {task}
Summary ranges: {segment_list_text}
For EACH range [0 → k], write ONE paragraph in first person. Describe what happened and which part of the task is completed by that point.
Table 9: Stage 2: Current Observation Analysis
System Prompt
You are a controlled reasoning module. Describe the indoor scene layout (open paths, obstacles, landmarks) based on the attached front, left, back, and right views. Only describe visible objects. Do NOT mention targets, goals, history, or next steps.
Output Format
"Analyze the current observations": [your detailed description]
Table 10: Stage 3: Future Action Reasoning (Infer Next Step)
System Prompt
IRON RULE: IF gt_action == "stop", declare arrival. ELSE, reason step-by-step how the move advances the task. The final sentence MUST name the target view and an object-defined region. FORBIDDEN: coordinates, pixels, red circles, proportions.
Target Guidance: {target_guidance} (Use this to locate the region, but do NOT mention the guidance itself).
Table 11: Stage 4: CoT Fusion and Polish
System Prompt
Rewrite the three-stage reasoning into a single, coherent English paragraph. Preserve spatial logic and the target view. Remove redundancies. Do NOT include stage titles. Output ONLY the polished paragraph.
Why it matters
This shows a practical path to deploying vision-language AI on real robots without retraining them to understand awkward 3D geometry, using far less training data than prior methods. The selective reasoning design, thinking only when needed, is a useful blueprint for any real-time robotic system that must balance speed and accuracy.
Terms in this paper
VLM (vision-language model) · an AI model trained to understand both images and text together
Chain-of-Thought (CoT) · a technique where the model writes out step-by-step reasoning before giving an answer
SLAM · a technology letting a robot map its surroundings while tracking its own position
GRPO · a reinforcement learning method that compares a group of candidate actions to reinforce the relatively better ones
SPL · a navigation metric measuring success weighted by how efficient the path taken was
Figures we cannot republish
Figure 1: The architecture of TAMP-Nav. Given visual-textual inputs, the agent compresses long-horizon history into an Anchor-Trajectory Memory, autonomously triggering reasoning at critical nodes. As a visual pointer, it selects the optimal multi-camera view and predicts a 2D pixel, which is projected into 3D for SLAM execution. Finally, the policy is optimized via Two-Level GRPO.
Figure 2: The Two-Level GRPO paradigm. TAMP-Nav superimposes trajectory-level rollouts (competing for global navigation success) with step-level candidate rollouts (exploring diverse 2D visual-spatial actions via VLM temperature sampling).
Figure 3: Spatial Heatmap of Reasoning (CoT) Triggers. Comparison between the SFT model (left) and the RL-aligned TAMP-Nav (right).
Figure 4: Performance distribution on long-horizon navigation tasks. The bars show the trajectory count in each length bin, and the curves show the Success Rate (SR) of the evaluated methods as trajectory length increases.
Figure 5: Learning curves of success rewards under different GRPO configurations. The plot illustrates the training progress over 800 steps for three variants: the full Two-Level GRPO, the version without annealed guided sampling (“without guidance”), and the standard GRPO using only trajectory-level rewards (“only trajectory reward”).
Figure 6: Visualization of real-world experiments
Figure 9: Instruction: “Find the red bag”
Figure 15: Visualization of STI token. We employ a clustering algorithm (23) to map high-dimensional embeddings into a 3D space.
Figure 16: Visualization of a simulated navigation trajectory. The agent successfully executes the complex instruction: “Go up the stairs and turn left. Wait at the doorway to the bedroom straight ahead.” by triggering selective reasoning at key decision points and outputting point-and-click actions.
Although Large Vision-Language Models (VLMs) have significantly advanced embodied navigation, their direct deployment remains challenging, as existing methods often force VLMs into unnatural action spaces that misalign with their 2D pre-training priors, compounded by rigid reasoning schedules and inefficient memory management. To overcome these limitations, we propose TAMP-Nav, a unified framework for efficient embodied navigation. First, we introduce a Pixel-to-3D Action Formulation (Point) that reformulates navigation into 2D visual prompting. Specifically, the VLM merely selects 2D pixels, which are then projected into 3D coordinates for a low-level SLAM controller. This design naturally aligns embodied execution with the VLM's inherent 2D visual capabilities. Second, we propose an integrated Selective Reasoning and Anchor-Trajectory Memory mechanism (Think and Memorize), which dynamically triggers Chain-of-Thought and retains high-fidelity memory only at critical nodes, compressing redundant trajectories into lightweight Space-Time Indicators, thereby preserving critical historical information and enhancing spatio-temporal perception. Finally, we design an efficient Two-Level Alignment Paradigm (Align) via Group Relative Policy Optimization (GRPO). By superimposing global outcome rewards with fine-grained process rewards, this dense supervision tightly aligns the agent's cognitive planning with physical environmental feedback, endowing the model with adaptive reasoning capabilities. Experiments demonstrate that TAMP-Nav achieves state-of-the-art performance (e.g., 66.2% SR on R2R-CE) with high runtime and sample efficiency (requiring only 90k training trajectories).
