Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

arXiv:2608.165902026-08-16

Zetta lets robots catch and fix their own mistakes in real time, then keep getting better

Zetta keeps a robot's core AI controller frozen and instead evolves a surrounding 'harness' of real-time watchdog code and recovery behaviors that improves over repeated attempts. Failed attempts are automatically clustered, diagnosed, and turned into reusable fix-it skills, backed by a purpose-built execution infrastructure called Z-Infra. This pushed success rates to 90.8% on LIBERO-Pro and 93.6% on RoboCasa, with 11.1x faster inference.

METAL MEDIA explanatory visual

Zetta lets robots catch and fix their own mistakes in real time, then keep getting better

01Existing embodied agent systems mostly reflect only after a whole task attempt is finished, so they can't catch or correct problems (like a slipping object or a collision) while the robot is still moving.
02Zetta leaves the underlying robot-controlling AI model (VLA/WAM) completely untouched, and instead adds high-frequency 'critic' monitoring code plus 'recovery' actions on top of it to enable real-time oversight.
03It runs three loops at different speeds: one governs actions moment-to-moment, one proposes fixes after batches of attempts, and one only accepts fixes into permanent memory after they pass validation.
04Failures are automatically grouped by cause, then diagnosed top-down through layers (evaluation, critic, state, planning, recovery, parameters) to find and patch only the minimal necessary layer, avoiding overfitting.
05A separate execution infrastructure (Z-Infra) decouples the agent's logic from the underlying hardware (different GPUs, CPUs, simulators), boosting valid attempt throughput from 1.7 to 35.1 episodes per minute, a 20.6x improvement.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Existing embodied agent systems mostly reflect only after a whole task attempt is finished, so they can't catch or correct problems (like a slipping object or a collision) while the robot is still moving.
Zetta leaves the underlying robot-controlling AI model (VLA/WAM) completely untouched, and instead adds high-frequency 'critic' monitoring code plus 'recovery' actions on top of it to enable real-time oversight.
It runs three loops at different speeds: one governs actions moment-to-moment, one proposes fixes after batches of attempts, and one only accepts fixes into permanent memory after they pass validation.
Failures are automatically grouped by cause, then diagnosed top-down through layers (evaluation, critic, state, planning, recovery, parameters) to find and patch only the minimal necessary layer, avoiding overfitting.
A separate execution infrastructure (Z-Infra) decouples the agent's logic from the underlying hardware (different GPUs, CPUs, simulators), boosting valid attempt throughput from 1.7 to 35.1 episodes per minute, a 20.6x improvement.

Figure 1: Zetta closes the loop for embodied self-evolution. Frequent runtime critics trigger recoveries during execution, while verified failures are distilled into reusable critic and recovery skills across rollouts. A hardware-decoupled rollout layer scales this process across heterogeneous environments, models, CPUs, and GPUs. Zetta enables sustained same-task improvement, zero-shot skill transfer, robotic "Aha moment", and accelerated agent execution.

Figure 2: Overview of the Zetta evolutionary framework. The system operates in a continuous loop between online execution and offline evolution. (Left) Parallel Rollouts: Action policy executes tasks, monitored by an Evolvable Harness (ℋ) composed of Critics (C), Recovery (R), and Tools (T), all under the high-level adjudication of an Orchestrator Agent. Rollouts are categorized into success and failure trajectories. (Right) Reflection & Evolve: Failed trajectories trigger a three-phase offline evolution cycle. Phase I profiles failures by clustering them against successful reference baselines. Phase II performs causal diagnosis to locate the root failure layer (L∗), followed by minimal harness repair to update C, R, and T. Phase III generalizes seed-specific repairs into a unified, versioned Harness Package (ℋmerged), which is then fed back into the execution loop.

Table 1: Core API primitives exposed to agents. All primitives support batching across sessions.
API Primitive	Functionality
Session Management — create once, run many episodes, close when done
create_sessions(requests)	Batch create sessions with env_family, env_config, lease_seconds. Returns SessionHandle with session_id.
renew_sessions(session_ids)	Extend lease duration for active sessions.
close_sessions(session_ids)	Close sessions and release resources (idempotent).
Environment Control — direct interaction for custom control logic
reset(session_ids, reset_spec)	Start new episode with task_id, seed, instruction. Returns episode_id and initial observation.
observe(session_ids)	Read current observation without side effects.
action_step(session_ids, actions)	Execute actions, return observation, reward, terminated, truncated flags.
Policy Inference — integrated model serving and stepping
policy_step(session_ids, policy_req)	Atomic observe → inference → step. Returns step results with executed actions.
policy_infer(session_ids, policy_req)	Inference only (no stepping). Returns actions for agent post-processing.
run_episode(session_ids, episode_req)	Execute complete episodes in workers. Returns summary (steps, reward, stop_reason).

Figure 3: Three-layer architecture of the rollout infrastructure. The Control Plane routes agent requests to specialized Env Workers and Rollout Workers, which manage environment simulation and model inference respectively.

Figure 4: Physical-intelligence “Aha” moments on LIBERO-Pro. Similar to RoboCasa, v0 denotes the Pure-VLA baseline and v1 represents early symptomatic repairs (e.g., staging or local gates) that yield stagnant performance. The “Aha” at v2 occurs when the agent identifies and resolves the decisive physical bottleneck (e.g., grasp retention or semantic approach), triggering a sharp increase in execution reliability.

Table 2: Success rates (%) on 18 RoboCasa Atomic-Seen tasks. Task identifiers follow Appendix A. The best result for each task is shown in bold. “Avg.” is the macro-average over all 18 tasks.
Method	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg.
Pure VLA (GR00T)	78	74	78	58	48	62	72	74	70	73.56
Zetta	96	92	94	96	86	80	96	96	86	93.56
Method	T10	T11	T12	T13	T14	T15	T16	T17	T18	Avg.
Pure VLA (GR00T)	62	92	76	88	96	50	90	82	74	73.56
Zetta	94	98	94	94	100	100	94	96	92	93.56

Figure 5: Physical-intelligence “Aha” moments on RoboCasa. L2-v0 is the original Pure-VLA baseline, while L2-v1 is an intermediate internal version. The flat L2-v0–L2-v1 segments summarize a period where early candidate fixes overfit individual failures. Once the agent identifies the key physical bottleneck (e.g., EEF alignment or centered contact), L2-v2 produces a sharp success-rate increase.

Figure 6: Cumulative physical-intelligence scaling on LIBERO-Pro Goal. Average performance across ten tasks under Goal-T and Goal-S perturbations. Each point denotes a selected cumulative harness version. The base VLA is frozen throughout evolution.

Table 3: Success rates (%) on LIBERO-Pro. The best result for each task is shown in bold. “Average” is the macro-average over 10 tasks.
Setting	Method	Task 0	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9	Average
Goal (T)	π0.5	0.0	95.0	10.0	0.0	100.0	0.0	20.0	80.0	5.0	0.0	31.0
Zetta	80.0	100.0	95.0	80.0	100.0	100.0	95.0	95.0	80.0	100.0	92.5
Goal (S)	π0.5	0.0	60.0	0.0	45.0	0.0	0.0	0.0	100.0	100.0	75.0	38.0
Zetta	90.0	65.0	80.0	85.0	95.0	95.0	100.0	100.0	100.0	80.0	89.0
LIBERO-10 (T)	π0.5	5.0	95.0	95.0	0.0	0.0	80.0	85.0	75.0	65.0	0.0	50.0
Zetta	35.0	95.0	100.0	0.0	25.0	100.0	95.0	80.0	100.0	0.0	63.0
LIBERO-10 (S)	π0.5	0.0	35.0	0.0	0.0	5.0	50.0	0.0	0.0	0.0	0.0	9.0
Zetta	90.0	50.0	95.0	75.0	15.0	65.0	5.0	0.0	0.0	5.0	40.0

Figure 7: Cross-task scaling from the Goal-T8 source task. Three cumulative Critic–Recovery capabilities discovered on Goal-T8 are transferred to Goal-T2, Goal-T6, and Goal-S3. Curves report complete fixed-seed evaluations of each cumulative stack. Shaded regions denote Wilson 95% confidence intervals. The right-hand frames illustrate the corresponding failure and recovery mechanisms rather than additional transfer measurements.

Table 4: Mapping from compact identifiers to official RoboCasa task names.
ID	Official task name
T1	NavigateKitchen
T2	TurnOnMicrowave
T3	PickPlaceCounterToStove
T4	PickPlaceSinkToCounter
T5	PickPlaceDrawerToCounter
T6	PickPlaceCounterToCabinet
T7	PickPlaceToasterToCounter
T8	TurnOnSinkFaucet
T9	CoffeeSetupMug

Figure 8: Cross-task scaling from the Goal-S5 source task. Cumulative Critic–Recovery capabilities discovered on Goal-S5 are transferred to Goal-S3, Goal-S4, and Goal-S9. Within each task, all four arms use the same fixed seeds, policy RNGs, checkpoint, and execution budget. Shaded regions denote Wilson 95% confidence intervals.

Figure 9: Cumulative reflection scaling across 18 RoboCasa tasks. The macro-average success rate increases from 73.56% for the frozen parent harness to 78.71%, 84.85%, 90.54%, and 93.56% after four cumulative global repair rounds. Each checkpoint retains the previously validated critic, recovery, and tool capabilities.

Table 5: Mapping of LIBERO-Pro task identifiers used in the main text to the underlying LIBERO tasks. “Goal” corresponds to LIBERO-Goal and “Long” corresponds to LIBERO-10. The same task indices are used for both T and S perturbation settings.
ID	LIBERO-Goal	ID	LIBERO-10 (Long)
Task 0	open the middle drawer of the cabinet	Task 0	put both the alphabet soup and the tomato sauce in the basket
Task 1	put the bowl on the stove	Task 1	put both the cream cheese box and the butter in the basket
Task 2	put the wine bottle on top of the cabinet	Task 2	turn on the stove and put the moka pot on it
Task 3	open the top drawer and put the bowl inside	Task 3	put the black bowl in the bottom drawer of the cabinet and close it
Task 4	put the bowl on top of the cabinet	Task 4	put the white mug on the left plate and put the yellow and white mug on the right plate
Task 5	push the plate to the front of the stove	Task 5	pick up the book and place it in the back compartment of the caddy
Task 6	put the cream cheese in the bowl	Task 6	put the white mug on the plate and put the chocolate pudding to the right of the plate
Task 7	turn on the stove	Task 7	put both the alphabet soup and the cream cheese box in the basket
Task 8	put the bowl on the plate	Task 8	put both moka pots on the stove
Task 9	put the wine bottle on the rack	Task 9	put the yellow and white mug in the microwave and close it

Figure 10: Reflection-driven scaling and transfer on PnP tasks. On PnP-Stove, successive rounds add object-relative pregrasp alignment, bounded regrasp after grasp loss, and stable placement. The lower panels apply the cumulative checkpoints to PnP-Sink, PnP-Cabinet, and PnP-Toaster without an additional training or evolution loop. The right-hand panels show the failure signatures identified by the critic and the corresponding recoveries.

Figure 11: Reflection-driven scaling and transfer on articulated interaction tasks. On TurnOffStove, successive rounds add target localization, collision-aware pre-contact approach, and stable EEF–target contact. The lower panels apply the cumulative checkpoints to TurnOnSinkFaucet, OpenCabinet, and TurnOnMicrowave without an additional training or evolution loop. The right-hand panels show the diagnosed target, approach, and contact failures together with the recovered states.

Why it matters

Today's robot AI models are brittle because training data is limited, and this work shows that success rates can be pushed much higher without retraining the model at all, just by evolving how it's monitored and corrected during execution. This offers a practical, cheaper path to more reliable robots, and skills learned on one task can transfer directly to similar tasks without extra training.

Terms in this paper

VLA (Vision-Language-Action model) · An AI model that takes in camera images and language instructions and directly outputs robot movements
rollout · One complete attempt by a robot at performing a task, from start to finish
closed-loop vs open-loop · Closed-loop means continuously checking and reacting during execution; open-loop means following a fixed plan and only checking afterward
critic · A code module that watches the robot's execution in real time and flags signs of trouble
recovery skill · An action taken to undo or retry a step once a problem has been detected

Original abstract (English)

Embodied agents are increasingly used to close the gap left by end-to-end policy models. Yet the agentic path has not realized closed-loop learning in physical execution: existing harnesses remain largely open-loop, following fixed skills during rollout and reflecting only after an episode completes. Such post-hoc reflection cannot govern execution as it unfolds, because physical interaction requires decisions to track rapidly changing robot-environment states at a frequency beyond today's large agentic models. We present Zetta, a closed-loop embodied harness that evolves code-based runtime critics and recovery skills online while keeping the base policy frozen. Through three timescale-separated loops, Zetta provides action-frequency governance, rollout-level critic-recovery proposal, and validation-gated skill updates. Together with Z-Infra, a rollout infrastructure decoupling agent logic from heterogeneous execution resources, Zetta achieves state-of-the-art success on LIBERO-Pro and RoboCasa under our current rollout budget, reaching 90.8% and 93.6%, with an 11.1x inference speedup; success continues to scale with self-exploration experience; learned skills transfer zero-shot, and clear robotic "Aha Moments" emerge. These results show that closed-loop harness self-evolution opens a scaling path for reliable physical intelligence.

Authors · Xin Ding

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Xin Ding et al., arXiv:2608.16590, CC BY 4.0