컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다

arXiv:2608.198912026-08-21

EXIMO: VLM Guided Exploration of VLA Policies

로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다

로봇 조작 정책(VLA)은 사람이 조종해서 만든 데이터로 학습하는데, 새로운 작업을 배우려면 다시 수백 시간의 사람 조작 데이터가 필요해서 비효율적이다. EXIMO는 언어 이해에 강한 비전-언어 모델(VLM)이 어려운 작업을 잘게 쪼개 로봇에게 지시하며 데이터를 모으고, 그 데이터로 로봇 모델을 다시 학습시킨 뒤 온라인 강화학습으로 마무리하는 3단계 방법이다. 22가지 조작 과제 실험에서 기존 방식보다 적은 데이터로 더 높은 성공률을 보였다.

METAL MEDIA 해설 도표

로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다

  1. 01구글의 Gemini Robotics On-Device(GROD)라는 30억 파라미터 로봇 정책을 기본 모델로 사용하고, Gemini라는 VLM이 감독관 역할을 맡아 긴 작업을 짧은 지시로 쪼개 로봇에게 실시간으로 명령한다
  2. 02VLM이 지시한 대로 로봇이 실행해 성공한 시도만 모아 데이터셋을 만들고, 이 데이터로 원래 VLA 모델을 다시 지도학습(파인튜닝)시켜 VLM 없이도 스스로 잘하게 만든다
  3. 03마지막으로 로봇 행동에 미세한 보정값을 더하는 잔차 정책을 강화학습으로 학습시켜 성능을 한 번 더 끌어올린다
  4. 0422개 조작 과제 실험에서 VLM 지시를 받은 로봇이 그냥 로봇보다 성공률이 높았고, 그 데이터로 다시 학습시킨 로봇은 VLM 지시를 받는 로봇보다도 더 잘했다
  5. 05강화학습까지 마친 EXIMO는 기존 로봇을 훨씬 많은 데이터로 강화학습시킨 경우보다도 더 높은 성공률과 빠른 성공 시간을 보였다
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 구글의 Gemini Robotics On-Device(GROD)라는 30억 파라미터 로봇 정책을 기본 모델로 사용하고, Gemini라는 VLM이 감독관 역할을 맡아 긴 작업을 짧은 지시로 쪼개 로봇에게 실시간으로 명령한다
  2. VLM이 지시한 대로 로봇이 실행해 성공한 시도만 모아 데이터셋을 만들고, 이 데이터로 원래 VLA 모델을 다시 지도학습(파인튜닝)시켜 VLM 없이도 스스로 잘하게 만든다
  3. 마지막으로 로봇 행동에 미세한 보정값을 더하는 잔차 정책을 강화학습으로 학습시켜 성능을 한 번 더 끌어올린다
  4. 22개 조작 과제 실험에서 VLM 지시를 받은 로봇이 그냥 로봇보다 성공률이 높았고, 그 데이터로 다시 학습시킨 로봇은 VLM 지시를 받는 로봇보다도 더 잘했다
  5. 강화학습까지 마친 EXIMO는 기존 로봇을 훨씬 많은 데이터로 강화학습시킨 경우보다도 더 높은 성공률과 빠른 성공 시간을 보였다
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 1: Example interaction of the VLM during the explore phase of Eximo. The VLM is given a sequence of images from the environment along with the task description in the prompt. The VLM analyzes the provided information within a <think> </think> block and provides an instruction in the <answer></answer> block for the VLA to execute.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 2: Success rate (top), time to success (middle), and episode length (bottom) of VLM orchestrated GROD, GROD with no orchestration, and GROD finetuned on the data collected from VLM orchestration. The baselines and the per-task legend (shared across all three plots) are shown on the right. Across the tasks, we observe that VLM orchestration achieves higher success rate than the model without orchestration. This illustrates the benefits of semantically guided exploration from the VLM. Furthermore, performing filtered SFT on the VLM orchestrated data gives additional performance gains in the success rate across the task, showcasing the advantages of distilling the VLM orchestrated trajectories into the VLA. The VLM orchestrated VLA also has a lower episode length and is therefore more data efficient. Note that time to success is only meaningful for tasks with non-zero success rate.
Table 1: Manipulation tasks used in our evaluation and their natural-language goals. The task identifiers (T2–T23) correspond to the per-task labels in Fig. 2 and Fig. 4; T1 in those plots denotes the average over all tasks. Reasoning variants replace the explicit object name with a semantic description that the agent must ground to the correct object, while the left/right caddy tasks additionally require spatial understanding.
IDTaskNatural-language goal
T2BowlGlassOnRackput the bowl and glass on the rack
T3BananaInBowl-Reasoning0put the item that a monkey can eat into the bowl
T4MugOnPlateput the mug on the plate
T5MugOnPlate-Reasoning0put the object you pour coffee in on the plate
T6MugOnPlate-Reasoning1put the object with a handle on top of the flat object
T7PenInContainerput the pen into the white container
T8PenInContainer-Reasoning0put the object you use to write into the white container
T9PenInContainer-Reasoning1put the thinnest object into the white container
T10CanOpenerInCaddy-Left-Reasoning0place the can opener in the left compartment of the caddy
T11CanOpenerInCaddy-Right-Reasoning0place the can opener in the right compartment of the caddy
T12MagnifierCanOpenerInCaddyput the magnifier and can opener in the caddy
T13MagnifierInCaddy-Left-Reasoning0place the magnifier in the left compartment of the caddy
T14MagnifierInCaddy-Right-Reasoning0place the magnifier in the right compartment of the caddy
T15ScissorsInCaddy-Left-Reasoning0place the scissors in the left compartment of the caddy
T16ScissorsInCaddy-Right-Reasoning0place the scissors in the right compartment of the caddy
T17ScissorsMagnifierInCaddyput the scissors and magnifier in the caddy
T18ScissorsScrewdriverInCaddyput the scissors and screwdriver in the caddy
T19ScrewdriverInCaddy-Left-Reasoning0place the screwdriver in the left compartment of the caddy
T20ScrewdriverInCaddy-Right-Reasoning0place the screwdriver in the right compartment of the caddy
T21ScrewdriverMagnifierInCaddyput the screwdriver and magnifier in the caddy
T22PlateBowlOnRackput the plate and bowl on the rack
T23PlateGlassOnRackput the plate and glass on the rack
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 3: Online RL performance of GROD + SFT and the base GROD model averaged across twenty tasks. We run the base GROD model for more timesteps to compensate for the additional data collected via VLM orchestration for the SFT. Due to the finetuning on VLM orchestrated data, GROD + SFT starts with a higher success rate. The model also converges to higher success rates and the base GROD model does not achieve the same performance, despite running online RL for significantly more environment steps. We observe the same behaviour for time-to-success. The performance is averaged across five seeds and we report the mean with two standard errors. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.
Figure 4: Success rate and time to success post RL finetuning. We compare the performance of the VLA after SFT on VLM orchestrated data, after SFT + RL and the base GROD model after RL. Across all tasks, we observe that the model after SFT + RL outperforms the other two baselines. Note that time to success is only meaningful for tasks with non-zero success rate.

왜 중요한가

사람이 로봇을 직접 조종해 데이터를 모으는 작업은 비용이 크고 시간이 오래 걸리는데, 이 방법은 말 잘하는 AI의 지식을 빌려 그 부담을 크게 줄인다. 새로운 로봇 작업을 훨씬 적은 인력과 시간으로 가르칠 수 있는 실용적인 방향을 보여준다.

로봇 팔에게 사람의 시연 없이 새 일 시키기, 말 잘하는 AI가 대신 가르친다 figure 4
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.
Figure 5: Performance of VLM orchestrated GROD with free form orchestration and only Pick&Place commands. The former commands the model in natural language, whereas the latter only provides pick and place commands. We find that GROD benefits from the VLM orchestrators and is robust to the types of orchestration, performing on par for both free form and Pick&Place orchestration.

이 논문의 용어

  • VLA(비전-언어-행동 모델) · 카메라로 본 장면과 언어 명령을 입력받아 로봇의 움직임(행동)을 출력하는 대형 모델
  • VLM(비전-언어 모델) · 이미지와 텍스트를 함께 이해하고 답할 수 있는 AI 모델. 여기서는 감독관 역할
  • 행동 복제(Behaviour Cloning) · 사람이 조종한 데이터를 그대로 따라 하도록 모델을 학습시키는 방식
  • 강화학습(RL) · 로봇이 스스로 시행착오를 겪으며 성공/실패 신호로 점점 개선하는 학습 방식
  • 잔차 정책(Residual Policy) · 기존 로봇 모델의 행동에 작은 보정값을 더해 성능을 개선하는 추가 학습 모델
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 6: Performance of VLM to residual policy distillation using AWBC for offline RL and residual RL during the online phase. Left: Performance during offline RL as a function of the offline dataset size. Right: Performance during the online RL phase. We observe that while the residual policy learns during the offline RL phase, it fails to benefit from it during the online RL one. This is most likely due to the distribution shift between the VLM orchestrated data (offline) and online rollouts. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.
Figure 7: Performance of VLM to residual policy distillation where the VLM is only used with probability p during the rollouts. Top row: We observe that not using the VLM yields the best performance across the tasks during evaluation, despite the VLM orchestration leading to significantly more success in the initial phases of data collection (bottom row). We believe this is because the residual policy cannot benefit from the VLM orchestrated tuples. These results are obtained across three seeds and the mean performance with standard deviation is reported.

저자 · Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: Bhavya Sukhija et al., arXiv:2608.19891, CC BY 4.0