컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

로봇이 움직이는 동안 실시간으로 스스로 실수를 고치고 배우는 시스템, Zetta

arXiv:2608.165902026-08-16

Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

로봇이 움직이는 동안 실시간으로 스스로 실수를 고치고 배우는 시스템, Zetta

로봇을 조종하는 AI(정책 모델)는 그대로 둔 채, 그 위에서 실시간으로 감시하고 실수를 고치는 '하네스'라는 감독 장치를 계속 진화시켜 성공률을 끌어올리는 Zetta를 제안했다. 실패한 시도들을 자동으로 분석해서 재사용 가능한 감시 규칙과 복구 동작을 만들고, 이를 뒷받침하는 전용 실행 인프라 Z-Infra도 함께 만들었다. 그 결과 LIBERO-Pro에서 90.8%, RoboCasa에서 93.6% 성공률을 달성했고 추론 속도도 11.1배 빨라졌다.

METAL MEDIA 해설 도표

로봇이 움직이는 동안 실시간으로 스스로 실수를 고치고 배우는 시스템, Zetta

  1. 01기존 로봇 에이전트들은 한 에피소드(작업 시도)가 끝난 뒤에야 되돌아보고 반성하는 방식이라, 로봇이 움직이는 도중 실시간으로 벌어지는 문제(물체 미끄러짐, 충돌 등)에 대응하지 못한다는 문제를 짚었다.
  2. 02Zetta는 로봇을 조종하는 기반 AI 모델(VLA/WAM)은 전혀 건드리지 않고, 그 위에 '비평가(Critic)'라는 고빈도 감시 코드와 '복구 스킬(Recovery)'이라는 대응 동작을 붙여서 실시간 감독을 가능하게 했다.
  3. 03속도가 다른 세 개의 루프(행동 단위 감독, 여러 시도 묶음 단위 후보 개선, 검증을 통과한 것만 반영하는 업데이트)를 두어 안전하게 진화하도록 설계했다.
  4. 04실패 사례를 자동으로 원인별로 묶고, 어느 계층(평가/감시/상태인식/계획/복구/파라미터 중)에서 문제가 생겼는지 상위 계층부터 순서대로 진단해서 꼭 필요한 최소한의 부분만 고치는 방식을 썼다.
  5. 05이기종 하드웨어(다른 종류의 GPU, CPU, 시뮬레이터)에서도 같은 에이전트 로직을 그대로 돌릴 수 있게 실행 인프라(Z-Infra)를 분리 설계해, 유효 롤아웃 처리량을 분당 1.7건에서 35.1건으로 20.6배 늘렸다.
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 기존 로봇 에이전트들은 한 에피소드(작업 시도)가 끝난 뒤에야 되돌아보고 반성하는 방식이라, 로봇이 움직이는 도중 실시간으로 벌어지는 문제(물체 미끄러짐, 충돌 등)에 대응하지 못한다는 문제를 짚었다.
  2. Zetta는 로봇을 조종하는 기반 AI 모델(VLA/WAM)은 전혀 건드리지 않고, 그 위에 '비평가(Critic)'라는 고빈도 감시 코드와 '복구 스킬(Recovery)'이라는 대응 동작을 붙여서 실시간 감독을 가능하게 했다.
  3. 속도가 다른 세 개의 루프(행동 단위 감독, 여러 시도 묶음 단위 후보 개선, 검증을 통과한 것만 반영하는 업데이트)를 두어 안전하게 진화하도록 설계했다.
  4. 실패 사례를 자동으로 원인별로 묶고, 어느 계층(평가/감시/상태인식/계획/복구/파라미터 중)에서 문제가 생겼는지 상위 계층부터 순서대로 진단해서 꼭 필요한 최소한의 부분만 고치는 방식을 썼다.
  5. 이기종 하드웨어(다른 종류의 GPU, CPU, 시뮬레이터)에서도 같은 에이전트 로직을 그대로 돌릴 수 있게 실행 인프라(Z-Infra)를 분리 설계해, 유효 롤아웃 처리량을 분당 1.7건에서 35.1건으로 20.6배 늘렸다.
Figure 1: Zetta closes the loop for embodied self-evolution. Frequent runtime critics trigger recoveries during execution, while verified failures are distilled into reusable critic and recovery skills across rollouts. A hardware-decoupled rollout layer scales this process across heterogeneous environments, models, CPUs, and GPUs. Zetta enables sustained same-task improvement, zero-shot skill transfer, robotic "Aha moment", and accelerated agent execution.
Figure 1: Zetta closes the loop for embodied self-evolution. Frequent runtime critics trigger recoveries during execution, while verified failures are distilled into reusable critic and recovery skills across rollouts. A hardware-decoupled rollout layer scales this process across heterogeneous environments, models, CPUs, and GPUs. Zetta enables sustained same-task improvement, zero-shot skill transfer, robotic "Aha moment", and accelerated agent execution.
Figure 2: Overview of the Zetta evolutionary framework. The system operates in a continuous loop between online execution and offline evolution. (Left) Parallel Rollouts: Action policy executes tasks, monitored by an Evolvable Harness (ℋ) composed of Critics (C), Recovery (R), and Tools (T), all under the high-level adjudication of an Orchestrator Agent. Rollouts are categorized into success and failure trajectories. (Right) Reflection & Evolve: Failed trajectories trigger a three-phase offline evolution cycle. Phase I profiles failures by clustering them against successful reference baselines. Phase II performs causal diagnosis to locate the root failure layer (L∗), followed by minimal harness repair to update C, R, and T. Phase III generalizes seed-specific repairs into a unified, versioned Harness Package (ℋm​e​r​g​e​d), which is then fed back into the execution loop.
Figure 2: Overview of the Zetta evolutionary framework. The system operates in a continuous loop between online execution and offline evolution. (Left) Parallel Rollouts: Action policy executes tasks, monitored by an Evolvable Harness (ℋ) composed of Critics (C), Recovery (R), and Tools (T), all under the high-level adjudication of an Orchestrator Agent. Rollouts are categorized into success and failure trajectories. (Right) Reflection & Evolve: Failed trajectories trigger a three-phase offline evolution cycle. Phase I profiles failures by clustering them against successful reference baselines. Phase II performs causal diagnosis to locate the root failure layer (L∗), followed by minimal harness repair to update C, R, and T. Phase III generalizes seed-specific repairs into a unified, versioned Harness Package (ℋm​e​r​g​e​d), which is then fed back into the execution loop.
Table 1: Core API primitives exposed to agents. All primitives support batching across sessions.
API PrimitiveFunctionality
Session Management — create once, run many episodes, close when done
create_sessions(requests)Batch create sessions with env_family, env_config, lease_seconds. Returns SessionHandle with session_id.
renew_sessions(session_ids)Extend lease duration for active sessions.
close_sessions(session_ids)Close sessions and release resources (idempotent).
Environment Control — direct interaction for custom control logic
reset(session_ids, reset_spec)Start new episode with task_id, seed, instruction. Returns episode_id and initial observation.
observe(session_ids)Read current observation without side effects.
action_step(session_ids, actions)Execute actions, return observation, reward, terminated, truncated flags.
Policy Inference — integrated model serving and stepping
policy_step(session_ids, policy_req)Atomic observe → inference → step. Returns step results with executed actions.
policy_infer(session_ids, policy_req)Inference only (no stepping). Returns actions for agent post-processing.
run_episode(session_ids, episode_req)Execute complete episodes in workers. Returns summary (steps, reward, stop_reason).
Figure 3: Three-layer architecture of the rollout infrastructure. The Control Plane routes agent requests to specialized Env Workers and Rollout Workers, which manage environment simulation and model inference respectively.
Figure 3: Three-layer architecture of the rollout infrastructure. The Control Plane routes agent requests to specialized Env Workers and Rollout Workers, which manage environment simulation and model inference respectively.
Figure 4: Physical-intelligence “Aha” moments on LIBERO-Pro. Similar to RoboCasa, v0 denotes the Pure-VLA baseline and v1 represents early symptomatic repairs (e.g., staging or local gates) that yield stagnant performance. The “Aha” at v2 occurs when the agent identifies and resolves the decisive physical bottleneck (e.g., grasp retention or semantic approach), triggering a sharp increase in execution reliability.
Figure 4: Physical-intelligence “Aha” moments on LIBERO-Pro. Similar to RoboCasa, v0 denotes the Pure-VLA baseline and v1 represents early symptomatic repairs (e.g., staging or local gates) that yield stagnant performance. The “Aha” at v2 occurs when the agent identifies and resolves the decisive physical bottleneck (e.g., grasp retention or semantic approach), triggering a sharp increase in execution reliability.
Table 2: Success rates (%) on 18 RoboCasa Atomic-Seen tasks. Task identifiers follow Appendix A. The best result for each task is shown in bold. “Avg.” is the macro-average over all 18 tasks.
MethodT1T2T3T4T5T6T7T8T9Avg.
Pure VLA (GR00T)78747858486272747073.56
Zetta96929496868096968693.56
MethodT10T11T12T13T14T15T16T17T18Avg.
Pure VLA (GR00T)62927688965090827473.56
Zetta9498949410010094969293.56
Figure 5: Physical-intelligence “Aha” moments on RoboCasa. L2-v0 is the original Pure-VLA baseline, while L2-v1 is an intermediate internal version. The flat L2-v0–L2-v1 segments summarize a period where early candidate fixes overfit individual failures. Once the agent identifies the key physical bottleneck (e.g., EEF alignment or centered contact), L2-v2 produces a sharp success-rate increase.
Figure 5: Physical-intelligence “Aha” moments on RoboCasa. L2-v0 is the original Pure-VLA baseline, while L2-v1 is an intermediate internal version. The flat L2-v0–L2-v1 segments summarize a period where early candidate fixes overfit individual failures. Once the agent identifies the key physical bottleneck (e.g., EEF alignment or centered contact), L2-v2 produces a sharp success-rate increase.
Figure 6: Cumulative physical-intelligence scaling on LIBERO-Pro Goal. Average performance across ten tasks under Goal-T and Goal-S perturbations. Each point denotes a selected cumulative harness version. The base VLA is frozen throughout evolution.
Figure 6: Cumulative physical-intelligence scaling on LIBERO-Pro Goal. Average performance across ten tasks under Goal-T and Goal-S perturbations. Each point denotes a selected cumulative harness version. The base VLA is frozen throughout evolution.
Table 3: Success rates (%) on LIBERO-Pro. The best result for each task is shown in bold. “Average” is the macro-average over 10 tasks.
SettingMethodTask 0Task 1Task 2Task 3Task 4Task 5Task 6Task 7Task 8Task 9Average
Goal (T)π0.50.095.010.00.0100.00.020.080.05.00.031.0
Zetta80.0100.095.080.0100.0100.095.095.080.0100.092.5
Goal (S)π0.50.060.00.045.00.00.00.0100.0100.075.038.0
Zetta90.065.080.085.095.095.0100.0100.0100.080.089.0
LIBERO-10 (T)π0.55.095.095.00.00.080.085.075.065.00.050.0
Zetta35.095.0100.00.025.0100.095.080.0100.00.063.0
LIBERO-10 (S)π0.50.035.00.00.05.050.00.00.00.00.09.0
Zetta90.050.095.075.015.065.05.00.00.05.040.0
(b) Swap/position-swap (S).
(b) Swap/position-swap (S).
Figure 7: Cross-task scaling from the Goal-T8 source task. Three cumulative Critic–Recovery capabilities discovered on Goal-T8 are transferred to Goal-T2, Goal-T6, and Goal-S3. Curves report complete fixed-seed evaluations of each cumulative stack. Shaded regions denote Wilson 95% confidence intervals. The right-hand frames illustrate the corresponding failure and recovery mechanisms rather than additional transfer measurements.
Figure 7: Cross-task scaling from the Goal-T8 source task. Three cumulative Critic–Recovery capabilities discovered on Goal-T8 are transferred to Goal-T2, Goal-T6, and Goal-S3. Curves report complete fixed-seed evaluations of each cumulative stack. Shaded regions denote Wilson 95% confidence intervals. The right-hand frames illustrate the corresponding failure and recovery mechanisms rather than additional transfer measurements.
Table 4: Mapping from compact identifiers to official RoboCasa task names.
IDOfficial task name
T1NavigateKitchen
T2TurnOnMicrowave
T3PickPlaceCounterToStove
T4PickPlaceSinkToCounter
T5PickPlaceDrawerToCounter
T6PickPlaceCounterToCabinet
T7PickPlaceToasterToCounter
T8TurnOnSinkFaucet
T9CoffeeSetupMug
Figure 8: Cross-task scaling from the Goal-S5 source task. Cumulative Critic–Recovery capabilities discovered on Goal-S5 are transferred to Goal-S3, Goal-S4, and Goal-S9. Within each task, all four arms use the same fixed seeds, policy RNGs, checkpoint, and execution budget. Shaded regions denote Wilson 95% confidence intervals.
Figure 8: Cross-task scaling from the Goal-S5 source task. Cumulative Critic–Recovery capabilities discovered on Goal-S5 are transferred to Goal-S3, Goal-S4, and Goal-S9. Within each task, all four arms use the same fixed seeds, policy RNGs, checkpoint, and execution budget. Shaded regions denote Wilson 95% confidence intervals.
Figure 9: Cumulative reflection scaling across 18 RoboCasa tasks. The macro-average success rate increases from 73.56% for the frozen parent harness to 78.71%, 84.85%, 90.54%, and 93.56% after four cumulative global repair rounds. Each checkpoint retains the previously validated critic, recovery, and tool capabilities.
Figure 9: Cumulative reflection scaling across 18 RoboCasa tasks. The macro-average success rate increases from 73.56% for the frozen parent harness to 78.71%, 84.85%, 90.54%, and 93.56% after four cumulative global repair rounds. Each checkpoint retains the previously validated critic, recovery, and tool capabilities.
Table 5: Mapping of LIBERO-Pro task identifiers used in the main text to the underlying LIBERO tasks. “Goal” corresponds to LIBERO-Goal and “Long” corresponds to LIBERO-10. The same task indices are used for both T and S perturbation settings.
IDLIBERO-GoalIDLIBERO-10 (Long)
Task 0open the middle drawer of the cabinetTask 0put both the alphabet soup and the tomato sauce in the basket
Task 1put the bowl on the stoveTask 1put both the cream cheese box and the butter in the basket
Task 2put the wine bottle on top of the cabinetTask 2turn on the stove and put the moka pot on it
Task 3open the top drawer and put the bowl insideTask 3put the black bowl in the bottom drawer of the cabinet and close it
Task 4put the bowl on top of the cabinetTask 4put the white mug on the left plate and put the yellow and white mug on the right plate
Task 5push the plate to the front of the stoveTask 5pick up the book and place it in the back compartment of the caddy
Task 6put the cream cheese in the bowlTask 6put the white mug on the plate and put the chocolate pudding to the right of the plate
Task 7turn on the stoveTask 7put both the alphabet soup and the cream cheese box in the basket
Task 8put the bowl on the plateTask 8put both moka pots on the stove
Task 9put the wine bottle on the rackTask 9put the yellow and white mug in the microwave and close it
Figure 10: Reflection-driven scaling and transfer on PnP tasks. On PnP-Stove, successive rounds add object-relative pregrasp alignment, bounded regrasp after grasp loss, and stable placement. The lower panels apply the cumulative checkpoints to PnP-Sink, PnP-Cabinet, and PnP-Toaster without an additional training or evolution loop. The right-hand panels show the failure signatures identified by the critic and the corresponding recoveries.
Figure 10: Reflection-driven scaling and transfer on PnP tasks. On PnP-Stove, successive rounds add object-relative pregrasp alignment, bounded regrasp after grasp loss, and stable placement. The lower panels apply the cumulative checkpoints to PnP-Sink, PnP-Cabinet, and PnP-Toaster without an additional training or evolution loop. The right-hand panels show the failure signatures identified by the critic and the corresponding recoveries.
Figure 11: Reflection-driven scaling and transfer on articulated interaction tasks. On TurnOffStove, successive rounds add target localization, collision-aware pre-contact approach, and stable EEF–target contact. The lower panels apply the cumulative checkpoints to TurnOnSinkFaucet, OpenCabinet, and TurnOnMicrowave without an additional training or evolution loop. The right-hand panels show the diagnosed target, approach, and contact failures together with the recovered states.
Figure 11: Reflection-driven scaling and transfer on articulated interaction tasks. On TurnOffStove, successive rounds add target localization, collision-aware pre-contact approach, and stable EEF–target contact. The lower panels apply the cumulative checkpoints to TurnOnSinkFaucet, OpenCabinet, and TurnOnMicrowave without an additional training or evolution loop. The right-hand panels show the diagnosed target, approach, and contact failures together with the recovered states.

왜 중요한가

지금의 로봇 AI는 데이터가 부족해 낯선 상황에서 쉽게 실패하는데, 이 연구는 모델을 재학습시키지 않고도 실행 중 감독과 복구 규칙만 진화시켜 성공률을 크게 높일 수 있음을 보여준다. 이는 로봇 학습에 드는 비용과 시간을 줄이면서도, 한 작업에서 배운 대처법을 비슷한 다른 작업에 그대로 재사용할 수 있는 실용적인 방향을 제시한다.

이 논문의 용어

  • VLA (Vision-Language-Action 모델) · 카메라로 본 것과 언어 명령을 받아 로봇의 움직임을 직접 출력하는 인공지능 모델
  • 롤아웃(rollout) · 로봇이 시작부터 끝까지 한 번 작업을 시도하는 과정 전체
  • closed-loop / open-loop(폐루프/개루프) · 폐루프는 실행 중 상황을 계속 확인하며 대응하는 방식, 개루프는 정해진 대로 실행만 하고 끝난 뒤에야 확인하는 방식
  • critic(비평가) · 로봇의 동작 상태를 실시간으로 감시해 문제 징후를 신고하는 코드 모듈
  • recovery skill(복구 스킬) · 감시 결과 문제가 감지됐을 때 상황을 되돌리거나 재시도하기 위한 대응 동작

저자 · Xin Ding

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: Xin Ding et al., arXiv:2608.16590, CC BY 4.0