τ_0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation

arXiv:2608.168852026-08-16

A robot that pauses to imagine several possible next moves before committing, instead of always deciding in one shot

τ0-VLA is a two-layer robot AI system that, when facing a hard decision during long chores like cleaning a room, cooking, or making milk tea, can imagine several candidate next steps, predict what the scene would look like after each, score them, and only then commit. It was trained on 40,115 hours of real-world robot data, and in physical robot tests, giving it more 'thinking' computation at test time improved both its next-step prediction accuracy and its actual task success rate. The paper is by Xiaowei Cai, posted on arXiv as 2608.16885.

METAL MEDIA explanatory visual

A robot that pauses to imagine several possible next moves before committing, instead of always deciding in one shot

01Most prior hierarchical robot AI systems pick the next subtask with a single forward pass, with no way to spend extra effort on hard or high-stakes decisions
02τ0-VLA only triggers extra search when the model is uncertain: it generates multiple candidate subtasks, uses a world model to predict the resulting image for each, scores them with a value model via beam search, then produces a final choice
03The chosen subtask is carried out by a low-level policy using a shared 40-dimensional action space, letting one model control multiple different robot bodies
04Across real-robot tasks such as room cleaning, ingredient prep, stir-fry cooking, milk tea making, laundry collection, and book organizing, more test-time computation led to both higher next-subtask prediction accuracy and higher closed-loop task success
05The improvement held even on out-of-distribution book arrangements never seen in training, showing the approach generalizes beyond its training data

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Most prior hierarchical robot AI systems pick the next subtask with a single forward pass, with no way to spend extra effort on hard or high-stakes decisions
τ0-VLA only triggers extra search when the model is uncertain: it generates multiple candidate subtasks, uses a world model to predict the resulting image for each, scores them with a value model via beam search, then produces a final choice
The chosen subtask is carried out by a low-level policy using a shared 40-dimensional action space, letting one model control multiple different robot bodies
Across real-robot tasks such as room cleaning, ingredient prep, stir-fry cooking, milk tea making, laundry collection, and book organizing, more test-time computation led to both higher next-subtask prediction accuracy and higher closed-loop task success
The improvement held even on out-of-distribution book arrangements never seen in training, showing the approach generalizes beyond its training data

Fig. 2: The hierarchical τ0-VLA architecture. (a) At a high-level inference step t, the proposal model P conditions on the latest multi-view observation ot, task instruction ℓ, carried execution memory ℳt−1, and previously generated subtask zt−1⋆. It produces observation-aligned memory ℳt and a direct proposal ztdir. (b) The low-level policy conditions on the generated subtask zt⋆, multi-view observation ot, proprioceptive state 𝐬t, and textual control metadata η. A vision-language backbone and Mixture-of-Transformers (MoT) action expert generate the action chunk 𝐚t:t+H−1 through conditional flow matching from a noisy action chunk. (c) On the TTC route, the proposal model is invoked N times for each retained branch to generate N candidates. Given the branch’s head-camera image and a candidate, the world model predicts the terminal head-camera image, and the value model assigns a candidate-quality score conditioned on the task instruction, candidate, and predicted image. Beam search globally retains the top-B branches by cumulative score and recursively expands them to depth D. The figure illustrates the root expansion with N=3 and B=2, where local and cumulative scores coincide. Deeper expansion is omitted for clarity. The reflective model then conditions on h¯t and the final branch summaries 𝒞t to generate the final subtask zt⋆. This output may coincide with a retained proposal but is not restricted to the retained set and is passed to the low-level policy for execution.

TABLE I: Long-horizon task performance. Each method–task setting uses 10 independently collected physical-robot trials. SR reports successful trials as x/10, and Progress is the normalized milestone-completion score. Avg. is the unweighted mean of the four task-level rates. The first four rows use direct execution. The final row uses the Hierarchical System with Plan Once and no beam search.
Method	Clean Room	Prepare Ingredients	Tomato and Egg Stir Fry	Make Milk Tea	Avg.
	SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑
GR00T N1.7 [26]	0/10	59.80%	1/10	68.57%	0/10	24.32%	0/10	28.46%	2.50%	45.29%
LingBot-VLA [40]	0/10	66.60%	0/10	35.00%	0/10	12.27%	0/10	63.85%	0.00%	44.43%
π0.5 [2]	4/10	86.20%	2/10	73.93%	0/10	49.77%	3/10	82.31%	22.50%	73.05%
τ0-VLA	4/10	92.80%	2/10	66.43%	0/10	65.00%	5/10	96.15%	27.50%	80.10%
τ0-VLA (Hierarchical System, Plan Once)	5/10	94.80%	4/10	82.86%	4/10	81.82%	5/10	91.92%	45.00%	87.85%

Fig. 3: Representative physical-robot evaluation tasks. (a) Clean Room requires collecting two dirty garments, placing them in a laundry basket, hanging a handbag, handing a blanket to a person, and disposing of table trash. (b) Prepare Ingredients requires retrieving a tomato and an egg, then cracking and stirring the egg while returning the tools. (c) Tomato and Egg Stir Fry requires chopping and transferring the tomatoes, cooking and seasoning the ingredients, plating the dish, and returning the cookware and utensils. (d) Make Milk Tea requires adding toppings, pouring milk and tea, sealing the cup, and inserting a straw. (e) Collect Laundry requires transferring a T-shirt from the bedside table to a laundry basket. (f) Tidy Makeup Table comprises three separately scored instruction-following groups that require different object selections and action sequences from matched visual states.

TABLE II: Direct-execution performance across embodiments. Tidy Makeup Table comprises three independently scored instruction-following groups. All methods execute the full task instruction without a high-level policy. Each task or task group is evaluated over 10 trials. SR reports successful trials as x/10, and Progress is the normalized milestone-completion score.
Method	Collect Laundry	Tidy Makeup Table
	T-shirt	Cotton Pad	Eyelash Curler	Makeup Puff
	SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑
GR00T N1.7 [26]	4/10	76.00%	10/10	87.50%	8/10	77.50%	7/10	52.50%
LingBot-VLA [40]	2/10	35.00%	9/10	67.50%	3/10	22.50%	3/10	33.75%
π0.5 [2]	9/10	88.00%	9/10	85.00%	8/10	85.00%	7/10	73.75%
τ0-VLA	10/10	97.00%	10/10	95.00%	9/10	92.50%	10/10	95.00%

Fig. 4: Next-subtask prediction accuracy under different high-level inference methods. We compare Plan Once, Best-of-N, and TTC (Ours) across four evaluation settings: Make Milk Tea, Book Organization (In-Domain), Book Organization (OOD), and Clean Room.

TABLE III: Closed-loop physical-robot performance with test-time computation. Each entry uses 10 independently collected trials. Book Organization uses shuffled initial arrangements and is reported without the in-domain and OOD split used in the open-loop evaluation. SR denotes task success rate, and Progress is the normalized milestone-completion score.
Method	Make Milk Tea	Book Organization	Clean Room
SR ↑	Progress ↑	SR ↑	Progress ↑	SR ↑	Progress ↑
Plan Once	5/10	91.92%	6/10	66.67%	5/10	94.80%
TTC	7/10	95.38%	9/10	93.33%	7/10	97.60%

Fig. 5: Relationship between computational cost and subtask-prediction accuracy. The accuracy increases with additional computation for the Make Milk Tea and Book Organization tasks, shown in panels (a) and (b), respectively. Each point represents an experimental result obtained at a different computational cost. The orange dashed curves show saturation fits to these results, and the gray dashed lines indicate the Plan Once baselines.

TABLE IV: Canonical 40-D state and action layout. Dimensions are one-indexed. For a rotation matrix 𝐑=[𝐫1,𝐫2,𝐫3], we use Rot6D⁡(𝐑)=[𝐫1⊤,𝐫2⊤]⊤.
Coordinates	Dimensions	State representation
Left EEF position	1–3	Cartesian position in meters
Left EEF orientation	4–9	Rot6D⁡(𝐑L)
Right EEF position	10–12	Cartesian position in meters
Right EEF orientation	13–18	Rot6D⁡(𝐑R)
Left gripper	19	native opening coordinate
Right gripper	20	native opening coordinate
Waist	21–22	two native coordinates
Planar base velocity	23–24	two native coordinates
Left arm joints	25–32	q1L,…,q8L in radians
Right arm joints	33–40	q1R,…,q8R in radians

TABLE V: Maximum duration of each physical-robot trial.
Task	Maximum duration
Clean Room	20 min
Prepare Ingredients	20 min
Tomato and Egg Stir Fry	20 min
Make Milk Tea	10 min
Book Organization	5 min
Collect Laundry	5 min
Tidy Makeup Table (each group)	5 min

TABLE VI: High-level instance families, synthesized by perturbing only the input memory while reading the corrected target from the demonstration. ℳn is the memory upon entering segment n. A single unified <think>/<memory>/<subtask> format instantiates all of them at zero extra annotation.
Family	Sampling position	Input → target memory	Target subtask	Deployment failure countered	Mix
within-subtask	anywhere in seg. n	ℳn→ℳn	seg. n	— (aligned, normal progression)	58%
transition	tail of seg. n	ℳn→ℳn+1	seg. n+1	starting a new subtask after completion	15%
catch-up	head of seg. n	ℳn−1→ℳn	seg. n	memory lag (behind the visual state)	10%
rollback	late in seg. n	ℳn+1…n+3→ℳn	retry seg. n	memory run-ahead (over-optimistic)	12%
error-think	annotated failure frame	ℳn→ type-dependent	recovery step	unnoticed execution failure	5%

Why it matters

In long robot tasks, picking the wrong subtask can doom the whole run even with perfect motor control, so a method that spends more thought only on hard decisions matters for building reliable household robots. It also shows that test-time scaling, an idea popularized in language models, can be transferred to physical robot control with real gains.

Terms in this paper

VLA (Vision-Language-Action) model · An AI model that turns camera views and language instructions into robot motor commands
Test-time computation · Spending extra computation at the moment of use, without retraining, to improve answer quality
World model · A model that predicts what the environment will look like after a given action
Beam search · A search method that keeps only the top-scoring few candidate paths at each step and expands them further
Execution memory · A running summary of how far the robot has progressed through a task so far

Original abstract (English)

Long-horizon robot manipulation requires a robot to both execute individual skills reliably and sequence them coherently over extended tasks. Most hierarchical vision-language-action (VLA) models make each such decision with a single forward pass, leaving no mechanism to allocate additional computation to difficult or consequential choices. We introduce τ_0-VLA, a hierarchical robot foundation model that formulates high-level subtask generation as a compute-scalable inference problem through world-model-guided test-time computation. At each inference step, the high-level policy uses execution memory to generate a subtask and, when needed, searches over alternatives before committing to its output. A low-level policy then executes the generated subtask across multiple robot embodiments. The policy is trained on 40,115 hours of heterogeneous real-world data with multimodal co-training. Across in-domain and distribution-shifted settings, allocating additional test-time computation substantially improves next-subtask prediction accuracy, and these gains translate into higher closed-loop success on long-horizon robot manipulation tasks.

Authors · Xiaowei Cai

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Xiaowei Cai et al., arXiv:2608.16885, arxiv-nonexclusive