Qwen-UI-Agent는 시뮬레이터가 아니라 실제 폰과 컴퓨터에서 화면 클릭과 명령줄 명령을 섞어 쓰며 여러 단계 작업을 끝내는 GUI 에이전트다
arXiv:2607.282272026-07-29
Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
Qwen-UI-Agent는 시뮬레이터가 아니라 실제 폰과 컴퓨터에서 화면 클릭과 명령줄 명령을 섞어 쓰며 여러 단계 작업을 끝내는 GUI 에이전트다
Qwen-UI-Agent는 모바일, 컴퓨터, 웹, 딥서치 환경을 하나로 묶어 화면 조작과 명령줄 실행을 함께 쓰는 GUI 에이전트다. 100대 넘는 실제 폰 환경과 대규모 시뮬레이터, 100턴 넘는 긴 작업을 학습하는 강화학습, 알림을 보고 먼저 움직이는 하네스 계층을 결합했다. 모바일 실기기 벤치마크에서 최고 성능을 냈고, 컴퓨터·웹·검색 과제에서도 최상위권 모델들과 견줄 만한 성적을 보였다.
METAL MEDIA 해설 도표
Qwen-UI-Agent의 네 가지 구성 요소
증거 상태측정 결과가 보고됨
환경 인프라모바일·컴퓨터·웹·딥서치를 위한 시뮬레이터와 실기기 100대 이상을 결합해 학습·평가 데이터를 만든다
데이터 플라이휠에이전트가 스스로 과제를 만들고 실패를 분석해 다음 학습 데이터를 자동으로 설계한다
통합 학습 프레임워크지도학습과 100턴 이상 긴 궤적에 대한 온라인 강화학습을 함께 적용해 모델을 훈련한다
하네스 계층알림 감지로 먼저 나서서 서비스를 시작하고, 모바일과 컴퓨터를 넘나드는 작업 상태를 유지한다
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
100대 이상의 실제 안드로이드 기기와 150개 넘는 앱으로 구성된 실기기 환경을 만들어 과제 설계, 데이터 수집, 강화학습, 평가에 모두 사용했다
화면 클릭 같은 GUI 동작, bash 명령줄 실행, 외부 API 호출, 사용자 확인 요청을 하나의 행동 공간으로 통합하고 한 번의 판단에 여러 행동을 묶어 실행하는 배치 실행을 지원한다
100턴이 넘는 긴 작업 궤적에 대해 검증기 기반 온라인 강화학습을 적용했고, 약 1만 개의 시뮬레이션 환경을 동시에 돌려 학습 속도를 높였다
에이전트가 스스로 과제를 만들고 실패를 분석해 다음 학습 데이터를 설계하는 오토리서치 방식의 데이터 플라이휠을 구축했다
휴대폰 알림(예: 항공편 취소)을 감지해 사용자 지시 없이도 대안을 찾아 승인받고 실행하는 사전 대응형 서비스 하네스를 만들었다
Figure 1: Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings.
Table 1: Action Space in Qwen-UI-Agent.
Action
Definition
GUI Actions
click
Clicks at coordinates (x,y).
double_click
Double-clicks at coordinates (x,y).
long_press
Long-presses at coordinates (x,y).
type
Types the specified text content.
open
Opens the specified app.
drag
Drags from start coordinates (x1,y1) to end coordinates (x2,y2).
system_button
Presses a system button, selected from back, home, menu, and enter.
wait
Waits for a specified duration in seconds.
CLI Actions
cli_command
Executes a bash command in the active CLI environment.
API Actions
api_call
Invokes an API with specified arguments.
Interaction and Control Actions
ask_user
Interact with the user to complete the task.
terminate
Ends the task with a final answer and a status of success or failed.
Figure 2: An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left panel summarizes the system capabilities underlying the trajectory, including cross-platform execution, a hybrid action space spanning GUI, CLI, and API operations, and environments ranging from large-scale sandboxes to real devices. The right panel illustrates a travel-recovery scenario triggered by a flight-cancellation notification. After identifying the affected tasks and commitments, the agent searches for alternative flights through an API, requests user approval before rebooking via a mobile GUI, updates the affected meeting schedule through desktop GUI and CLI actions, and sends the revised artifact to the relevant recipients.
Table 2: Performance comparison on the GUI-only subset of MobileWorld (Kong et al. 2026), consisting of 117 tasks. We include representative general-purpose VLMs and specialized GUI models. The best result is highlighted in bold, and the second-best result is underlined.
Model
Access / Size
Success Rate (%)
General-purpose VLMs
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
73.2
GPT-5.6 Sol (OpenAI 2026b)
Closed-source
70.1
Claude Opus 4.8 (Anthropic 2026)
Closed-source
67.5
Seed 2.0 Pro (ByteDance Seed 2026a)
Closed-source
63.2
Qwen 3.7 Plus (Qwen Team 2026a)
397B-A17B
62.3
Gemini 3.1 Pro (Google 2026a)
Closed-source
58.1
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
55.6
Specialized GUI Models
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
43.9
MAI-UI-235B-A22B (Zhou et al. 2025a)
235B-A22B
39.7
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
30B-A3B
17.1
Ours
Qwen-UI-Agent
27B
82.1
Qwen-UI-Agent
35B-A3B
65.0
Figure 3: The environment infrastructure of Qwen-UI-Agent. (a) Scalable sandbox environments spanning mobile-use, computer-use, browser-use, and DeepSearch provide controllability and repeatable evaluation for data synthesis and training. (b) A sim-to-real bridge extends agents to real devices with real applications, networks, and account states, supporting user takeover for login, payment, permission, and confirmation. (c) A hybrid GUI+CLI action space interleaves graphical operations with direct command execution. (d) A unified interface standardizes the thinking–action–observation loop across heterogeneous environments.
Table 3: Performance comparison on real-device mobile benchmarks. MobileWorld-Real is our proposed Chinese real-device mobile GUI benchmark (Section 3.2), and AndroidDaily (Sui et al. 2026) covers high-frequency daily scenarios on real Android devices.
Model
Access / Size
MobileWorld-Real
AndroidDaily
Baselines
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
88.7
95.2
Gemini 3.1 Pro (Google 2026a)
Closed-source
86.2
93.8
GPT-5.6 Sol (OpenAI 2026b)
Closed-source
85.4
92.6
Claude Opus 4.8 (Anthropic 2026)
Closed-source
84.7
93.0
Qwen 3.7 Plus (Qwen Team 2026a)
Closed-source
72.7
79.8
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
62.6
67.6
PhoneBuddy-4B (Tang et al. 2026)
4B
53.5
69.0
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
30B-A3B
33.0
61.7
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
32.4
60.9
GELab-Zero-4B-preview (Yan et al. 2025)
4B
31.3
73.4
Ours
Qwen-UI-Agent
27B
92.2
97.5
Qwen-UI-Agent
35B-A3B
87.4
93.9
Figure 4: Real-device mobile runtime with closed-loop environment governance. The health-aware scheduler routes each task to an eligible phone, App/account, and display; unhealthy targets remain blacklisted until repair and revalidation. Virtual displays allow one phone to run multiple Apps concurrently. Evidence-based review separates task success, model failure, and environment failure from the complete trajectory, and confirmed environment issues are fed back to the scheduler.
Table 4: Performance comparison on OSWorld-Verified (Xie et al. 2024).
Model
Access / Size
Success Rate (%)
Baselines
Claude Opus 4.8 (Anthropic 2026)
Closed-source
83.4
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
78.8
GPT-5.5 (OpenAI 2026a)
Closed-source
78.7
Gemini 3.5 Flash (Google 2026b)
Closed-source
78.4
Gemini 3.1 Pro (Google 2026a)
Closed-source
76.2
Qwen 3.7 Plus (Qwen Team 2026a)
Closed-source
73.3
MiniMax M3 (MiniMax 2026)
428B-A23B
75.2
Kimi K2.6 (Moonshot AI 2026)
1T-A32B
73.1
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
32B
56.5
Ours
Qwen-UI-Agent
27B
79.5
Figure 5: The data flywheel of Qwen-UI-Agent. Domain capability bootstrapping initializes training, iterative refinement loop identifies capability weaknesses and generate targeted tasks, and the resulting data improve the next training iteration.
Table 6: Performance comparison on WebArena (Zhou et al. 2024). Baseline results marked with ∗ are obtained from our own evaluation under the same experimental setup.
Model
Access / Size
Success Rate (%)
Baselines
Claude Opus 4.8∗ (Anthropic 2026)
Closed-source
71.9
GPT-5.5∗ (OpenAI 2026a)
Closed-source
69.5
Gemini 3.1 Pro∗ (DeepMind 2025)
Closed-source
65.3
Qwen 3.7 Plus∗ (Qwen Team 2026a)
Closed-source
59.0
CUA-GYM-A17B (Wang et al. 2026)
397B-A17B
56.0
Kimi K2.6∗ (Moonshot AI 2026)
1T-A32B
55.8
Qwen3.5-397B-A17B (Qwen Team 2026b)
397B-A17B
54.0
GUI-Owl-1.5-32B-Thinking (Xu et al. 2026)
32B
48.4
Qwen3.5-27B (Qwen Team 2026b)
27B
41.5
Qwen3.5-35B-A3B (Qwen Team 2026b)
35B-A3B
40.8
Ours
Qwen-UI-Agent
27B
73.6
Qwen-UI-Agent
35A3B
69.2
Human Performance (Zhou et al. 2024)
–
78.2
Figure 6: Overview of our harness for proactive service initiation and cross-platform execution. (I) Without the harness, users must manually interpret notifications, retrieve related context, and provide instructions for the agent. The proactive harness instead detects a flight cancellation, reasons over relevant affairs, proactively implement flight rebooking, railway alternatives, and meeting-conflict checks, and report an actionable plan. (II) The harness also maintains task state and support cross-platform execution. In the restaurant-selection example, Qwen-UI-Agent discovers candidates on mobile, organizes them in a desktop spreadsheet, sends the artifact for approval, and saves the selected locations back to mobile.
Table 7: Performance comparison on DeepSearch benchmarks: BrowseComp (BC) and BrowseComp-ZH (BC-ZH). “–” indicates the result is not reported.
Model
Access / Size
BC (%)
BC-ZH (%)
Baselines
GPT-5.5 (OpenAI 2026a)
Closed-source
90.1
–
Seed 2.1 Pro (ByteDance Seed 2026b)
Closed-source
86.2
–
Gemini 3.1 Pro (Google 2026a)
Closed-source
85.9
–
Claude Opus 4.8 (Anthropic 2026)
Closed-source
84.3
–
UI-TARS-2 (Wang et al. 2025a)
Closed-source
29.6
50.5
Qwen3.5-397B-A17B (Qwen Team 2026b)
397B-A17B
78.6
70.3
Apodex-1.0-mini (Apodex Team 2026)
35B-A3B
71.5
80.6
Qwen3.5-27B (Qwen Team 2026b)
27B
61.0
62.1
GLM-4.7 (Z.ai 2025)
358B
52.0
66.6
DeepSeek-V3.2 (DeepSeek-AI 2025)
685B
51.4
65.0
Tongyi-DR-30B (Tongyi DeepResearch 2025)
30B-A3B
43.4
46.7
Ours
Qwen-UI-Agent
27B
64.1
75.0
Figure 7: Overview of MobileWorld-Real, a real-device benchmark with human-written tasks that reflect the breadth of everyday mobile use. Representative examples and aggregate statistics show broad domain coverage and a long-tailed App distribution. Matched-model results further show lower success rates and longer trajectories than on AndroidDaily, highlighting the challenge of real-world mobile interaction.
Table 8: Performance comparison on GUI grounding benchmarks. ScreenSpot-Pro reports no-zoom results, with zoom-in results shown in parentheses when available. Baseline results marked with ∗ are obtained from our own evaluation.
Models
Grounding Benchmarks
SS-Pro(zoom in)
SS-V2
MM-GUI-L2
OSW-G-R
UI-Vision
Baselines
Qwen 3.7 Plus* (Qwen Team 2026a)
68.9 (79.0)
96.6
90.5
78.2
68.0
Seed 2.1 Pro* (ByteDance Seed 2026b)
65.3 (80.7)
96.6
90.9
78.0
62.0
Qwen3.5-4B* (Qwen Team 2026b)
59.1 (60.3)
94.3
83.4
62.0
32.6
Qwen3.5-35B-A3B* (Qwen Team 2026b)
64.5 (68.6)
95.2
87.5
63.6
52.3
Qwen3.5-27B* (Qwen Team 2026b)
68.4 (70.3)
96.1
89.1
67.9
46.6
GUI-Owl-1.5-2B-Instruct (Xu et al. 2026)
57.8 (70.4)
89.7
72.1
62.6
–
GUI-Owl-1.5-4B-Instruct (Xu et al. 2026)
66.8 (75.6)
93.2
83.2
68.4
–
GUI-Owl-1.5-8B-Instruct (Xu et al. 2026)
71.1 (77.8)
93.7
82.5
69.3
–
GUI-Owl-1.5-32B-Instruct (Xu et al. 2026)
72.9 (80.3)
95.3
86.8
69.7
–
UI-Venus-1.5-2B (Gao et al. 2026)
57.7 (64.6)
92.8
80.3
59.4
44.8
UI-Venus-1.5-8B (Gao et al. 2026)
68.4 (73.9)
95.9
88.1
69.7
46.5
UI-Venus-1.5-30B-A3B (Gao et al. 2026)
69.6 (74.8)
96.2
88.6
70.6
54.7
ZoomOnce-4B (Liu et al. 2026)
66.2
95.2
87.6
73.1
40.2
HyMobileAgent-A3B (Shen et al. 2026)
66.5
96.2
89.3
–
–
MAI-UI-2B (Zhou et al. 2025a)
57.4 (62.8)
92.5
82.6
63.5
30.3
MAI-UI-8B (Zhou et al. 2025a)
65.8 (70.9)
95.2
88.8
72.9
40.7
MAI-UI-32B (Zhou et al. 2025a)
67.9 (73.5)
96.5
91.3
75.0
47.1
GTA1-7B (Yang et al. 2025a)
50.1
92.4
78.5
67.7
–
GTA1-32B (Yang et al. 2025a)
63.6
95.2
83.4
72.2
–
UI-Ins-7B (Chen et al. 2025)
52.2
94.0
83.1
-
-
UI-Ins-32B (Chen et al. 2025)
57.0
94.9
84.9
-
-
OpenCUA-7B (Wang et al. 2025b)
50.0
92.3
–
–
29.7
OpenCUA-32B (Wang et al. 2025b)
55.3
93.4
–
–
33.3
OpenCUA-72B (Wang et al. 2025b)
60.8
92.9
–
–
37.3
Ours
Qwen-UI-Agent-4B
67.8 (74.0)
94.9
87.9
70.5
51.6
Qwen-UI-Agent-35B-A3B
76.1 (80.2)
96.7
92.0
74.6
65.9
Qwen-UI-Agent-27B
76.6 (81.5)
97.5
92.6
78.5
70.0
Figure 8: Demonstration of real-device mobile GUI execution. The trajectory is rendered as key frames with the executed action annotated beneath each frame. In this cross-app task, the agent looks up the target address on Amap, finds the most popular café nearby on Dianping, and posts a summary of the findings on RedNote.
Table 10: Failure-pattern distribution over all failed Qwen 3.7 Plus trajectories on real devices.
Dimension
Failure Pattern
Prop.
Typical Behavior
Execution Capability Limitations (40.3%)
Exploration Failure
19.5%
Fail to locate deep in-app entries
Erroneous Action Loops
14.3%
Repeat ineffective actions
Lost Execution State
6.5%
Forget finished sub-tasks
Real-world Scenario Challenges (52.0%)
UI Misreading
24.7%
Misread stateful page semantics
Pop-up Interference
18.2%
Ads, paywalls, CAPTCHAs, blank pages
Physical Widget Control
9.1%
Overshoot targets, never converge
Others
–
7.7%
Under-execution, premature stop
Figure 9: Demonstration of hybrid GUI+CLI execution in computer-use tasks. Key frames are shown with the emitted actions, where CLI commands are highlighted in blue, GUI actions in red, and each step carries a short annotation of its intent. In this example, the agent selects the target photo and filters by stitching all candidates into a single montage via CLI for one-shot visual inspection, rather than opening each image individually. It then leverages hybrid execution to combine CLI-based processing with GUI-based verification for the final output.
Table 11: GUI+CLI usage and batched-execution statistics on OSWorld-Verified and OSWorld-v2. Panel (a) reports CLI and batched-action usage at the action and task levels. Panel (b) reports the composition of batched outputs. Mean batch size counts the number of primitive actions in a batch. Differences are computed as OSWorld-v2 minus OSWorld-Verified. Differences are reported in percentage points (pp).
Statistic
Level
OSWorld-Verified
OSWorld-v2
Difference (pp)
(a) Overall CLI and batched-action usage
CLI
Action
40.7%
55.1%
+14.4
CLI
Task
92.0%
98.2%
+6.2
Batched
Action
39.6%
41.6%
+2.0
Batched
Task
62.1%
88.9%
+26.8
(b) Composition of batched outputs
GUI-only batches
75.8%
64.7%
-11.1
CLI-only batches
13.1%
15.0%
+1.9
Mixed GUI+CLI batches
11.0%
20.3%
+9.3
Mean primitive actions per batch
3.1
3.1
0.0
Figure 10: Demonstration of DeepSearch-assisted GUI execution. The upper panels show the multi-round search process, including keyword-based DeepSearch, targeted web fetches with intermediate thoughts, and the final search summary; the lower row shows the subsequent GUI trajectory. DeepSearch resolves the knowledge and reasoning problem before GUI execution, converting the vague cross-source request into an explicit target: Qwen-UI-Agent identifies the largest comeback in the World Cup knockout stage through DeepSearch, then opens RedNote and navigates directly to the highest-engagement related post from the past week without blind in-app searching.
Table 13: Performance comparison between frequent and long-tail actions.
Action Group
Original Data Proportion
Reward before Action RL
Reward after Action RL
Frequent actions
80.1%
88.3%
92.3%
Long-tail actions
19.9%
71.5%
77.9%
Figure 11: Demonstration of proactive service based on mobile notifications. The trajectory is organized into highlighted stages: proactive task initiation from a detected notification, proactive execution phases, and a decision-ready result, with the executed action annotated beneath each key frame. Flight-cancellation recovery: when the user’s next-morning flight is canceled and a 14:00 demo is at risk, the harness proactively searches alternative flights and high-speed trains, evaluates which options arrive on time, and presents a decision-ready travel recovery plan.
실제로 확인된 결과
MobileWorld-Real(실기기, 400여 개 과제)에서 92.2% 성공률로 Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, Seed 2.1 Pro를 각각 6.0, 7.5, 6.8, 3.5퍼센트포인트 앞섰다
AndroidDaily 실기기 벤치마크에서 97.5%, 시뮬레이션 기반 MobileWorld에서 82.1%로 Opus 4.8, GPT-5.6 Sol, Seed 2.1 Pro를 14.6, 12.0, 8.9퍼센트포인트 앞섰다
OSWorld-Verified(컴퓨터 사용)에서 79.5%로 GPT-5.5, Gemini 3.1 Pro, Seed 2.1 Pro보다 높았고, 더 어려운 OSWorld-v2에서는 부분진행 점수 40.0%로 MiniMax M3와 Qwen 3.7 Plus보다 각각 17.7, 18.5퍼센트포인트 높으면서 스텝 수는 58.4%, 21.7% 적었다
WebArena(웹 사용)에서 73.6%로 Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro를 1.7, 4.1, 8.3포인트 앞섰고, BrowseComp 64.1%·BrowseComp-ZH 75.0%로 Qwen3.5-397B-A17B와 UI-TARS-2를 앞섰다
OSWorld 계열 실행 궤적 분석에서 CLI 동작이 전체 행동의 40.7~55.1%를 차지했고 배치 행동이 39.6~41.6%를 차지하며 배치당 평균 3.1개 행동을 묶었다
Figure 12: Demonstration of cross-platform task execution. In this workflow, mobile subtasks run on virtual screens of the physical device, so execution does not block the user’s own actions. Parallel multi-app search: the agent searches sushi restaurants on Dianping, Meituan, and Amap concurrently through multiple virtual screens, consolidates their ratings, and summarizes the top three options in a local report opened on the computer.
어디에 쓸 수 있나
알림이나 상태 변화를 감지해 사용자 대신 대안을 찾고 승인만 받는 사전 대응형 모바일 비서 설계
화면 조작만으로는 비효율적인 파일 정리·데이터 처리 작업에 명령줄 실행을 함께 쓰는 데스크톱 자동화 도구 설계
여러 앱을 오가며 정보를 모으고 정리해 다른 기기로 전달하는 크로스플랫폼 업무 자동화 워크플로 설계
실제 기기에서 벌어지는 팝업, 플레이스홀더 텍스트, 슬라이더 조작 같은 문제를 다루는 에이전트 학습 데이터 설계 참고
한계와 남은 검증
보고된 실기기 벤치마크는 MobileWorld-Real과 AndroidDaily 두 가지이며 다른 언어권·기기 환경에서의 일반화는 확인되지 않았다
실패 분석은 Qwen 3.7 Plus라는 특정 비교 모델의 실기기 궤적을 대상으로 한 것으로 Qwen-UI-Agent 자체의 잔여 실패 유형은 상세히 제시되지 않았다
OSWorld-v2에서는 이항 성공률(13.9%)이 최고가 아니라 두 번째였고 부분진행 점수도 세 번째로, 가장 어려운 조건에서는 아직 1위가 아니다
사전 대응형 서비스 하네스의 장기적 사용자 반응 기반 자기개선 효과는 사례 시연 위주로 설명되어 있어 정량적 검증 결과는 제시되지 않았다
보고된 비교는 Opus 4.8, Gemini 3.1 Pro, GPT-5.6 Sol 등 특정 시점의 모델들과의 비교이며 이후 모델 갱신에 따른 상대적 우위 변화는 알 수 없다
왜 중요한가
GUI 에이전트가 데모나 시뮬레이터에서만 잘 작동하고 실제 기기에서는 자주 실패한다는 문제를 정면으로 다뤄, 실제 배포에 가까운 조건에서의 성능 격차를 좁히려는 시도다. 실제 앱을 다루는 자동화, 여러 기기를 넘나드는 업무 처리, 알림 기반 자동 대응 같은 서비스를 설계하는 사람들에게 참고할 구조와 수치를 제공한다.
