컴퓨터 화면을 직접 조작하는 AI 에이전트, 슬라이더나 드래그 앤 드롭 같은 작은 UI 부품 앞에서 자꾸 막힌다
arXiv:2608.183072026-08-20
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
컴퓨터 화면을 직접 조작하는 AI 에이전트, 슬라이더나 드래그 앤 드롭 같은 작은 UI 부품 앞에서 자꾸 막힌다
연구진은 웹 화면을 직접 보고 클릭·드래그하는 AI 에이전트를 평가하기 위해 ComponentBench라는 벤치마크를 만들었다. 날짜 선택기, 슬라이더, 드래그 앤 드롭 목록 같은 97가지 UI 부품 유형을 2,910개의 실제 작동하는 과제로 구현하고, 사람이 직접 수행한 기록과 비교해 성공 여부뿐 아니라 효율성까지 측정했다. 그 결과 같은 AI 모델이라도 화면을 어떻게 보여주고 어떻게 조작하게 하느냐에 따라 성공률이 30%포인트 넘게 달라졌고, 사람은 1~2번 클릭으로 끝내는 슬라이더나 창 분할 조작을 에이전트는 여전히 어려워했다.
METAL MEDIA 해설 도표
컴퓨터 화면을 직접 조작하는 AI 에이전트, 슬라이더나 드래그 앤 드롭 같은 작은 UI 부품 앞에서 자꾸 막힌다
01웹 화면의 개별 부품(버튼 묶음, 필터 가능한 표 등) 단위로 AI 에이전트를 진단하는 ComponentBench를 만들었다. Ant Design, MUI, Mantine 같은 실제 UI 라이브러리로 2,910개 과제를 구현하고, 사람이 두 번씩 수행한 기록 중 더 짧은 경로를 정답 기준으로 삼았다.
02GPT-5.4, Gemini 3 Flash, GPT-5 mini 등 7개 모델을 화면 인식 방식 4가지(접근성 트리 텍스트, 번호 오버레이, 픽셀 좌표만, DOM 정보를 함께 주는 Browser-Use)로 나눠 시험했다. 같은 GPT-5 mini 모델이 접근성 트리 방식에서는 83.1%를 통과했지만 픽셀 좌표만 주어졌을 때는 48.9%로 떨어졌다.
03가장 빠른 설정조차 사람보다 평균 3.7배 시간이 더 걸렸고, 슬라이더·드래그 앤 드롭·창 분할처럼 사람은 2번 안팎의 동작으로 끝내는 작업을 여러 에이전트가 60% 미만의 성공률로 힘겨워했다.
04화면이 복잡하거나(clutter) 요소 간격이 좁을 때 스크린샷만 보는 방식(Pixel)의 성능이 특히 크게 떨어졌고, DOM 정보를 활용하는 방식은 이런 시각적 혼잡에 거의 영향받지 않았다.
05실패 원인을 추적한 결과 에이전트는 목표를 이해하지 못해서가 아니라, 슬라이더 값을 정확히 맞추지 못하거나 팝업/편집 모드가 커밋 전에 사라지는 등 특정 조작 메커니즘 자체에서 반복적으로 막히는 경우가 많았다.
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
웹 화면의 개별 부품(버튼 묶음, 필터 가능한 표 등) 단위로 AI 에이전트를 진단하는 ComponentBench를 만들었다. Ant Design, MUI, Mantine 같은 실제 UI 라이브러리로 2,910개 과제를 구현하고, 사람이 두 번씩 수행한 기록 중 더 짧은 경로를 정답 기준으로 삼았다.
GPT-5.4, Gemini 3 Flash, GPT-5 mini 등 7개 모델을 화면 인식 방식 4가지(접근성 트리 텍스트, 번호 오버레이, 픽셀 좌표만, DOM 정보를 함께 주는 Browser-Use)로 나눠 시험했다. 같은 GPT-5 mini 모델이 접근성 트리 방식에서는 83.1%를 통과했지만 픽셀 좌표만 주어졌을 때는 48.9%로 떨어졌다.
가장 빠른 설정조차 사람보다 평균 3.7배 시간이 더 걸렸고, 슬라이더·드래그 앤 드롭·창 분할처럼 사람은 2번 안팎의 동작으로 끝내는 작업을 여러 에이전트가 60% 미만의 성공률로 힘겨워했다.
화면이 복잡하거나(clutter) 요소 간격이 좁을 때 스크린샷만 보는 방식(Pixel)의 성능이 특히 크게 떨어졌고, DOM 정보를 활용하는 방식은 이런 시각적 혼잡에 거의 영향받지 않았다.
실패 원인을 추적한 결과 에이전트는 목표를 이해하지 못해서가 아니라, 슬라이더 값을 정확히 맞추지 못하거나 팝업/편집 모드가 커밋 전에 사라지는 등 특정 조작 메커니즘 자체에서 반복적으로 막히는 경우가 많았다.
Table 1: ComponentBench-Full pass rates (%) by model and observation/action space. All runs use a 20-step budget. Browser-Use is a tool-rich reference regime with DOM access; AX-tree, SoM, and Pixel share the same harness and differ only in observation and action space. Task-resampling bootstrap 95% CIs (B=10,000) have half-width ≤1.9% per cell. †Native pixel-only harness with coordinate prediction. ‡Browser-Use served via Amazon Bedrock (precision unspecified); other Qwen cells use local vLLM FP8 (Appendix M).
Model
Browser-Use
AX-tree
SoM
Pixel
Gemini 3 Flash
95.2
89.6
87.1
85.4
GPT-5.4
90.4
81.5
77.0
83.8
Gemini 3.1 Flash-Lite
87.4
77.7
73.5
63.3
GPT-5 mini
87.0
83.1
78.5
48.9
GPT-5.4 mini
85.8
79.1
74.7
77.1
Qwen3-VL-235B‡
78.8
77.0
54.4
50.5
UI-TARS-1.5-7B†
—
—
—
12.6
Table 2: Step efficiency (% of 2,910 tasks). ≤H/≤2H: solved within 1/2× human steps, where H=max(human steps,1); hover-only references count as H=1. Browser-Use “steps” are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. †Native pixel-only harness. Full ≤3H/5H breakdown in Appendix C.
Browser-Use
AX-tree
SoM
Pixel
Model
Pass
≤H
≤2H
Pass
≤H
≤2H
Pass
≤H
≤2H
Pass
≤H
≤2H
Gemini 3 Flash
95.2
56.1
85.3
89.6
74.7
82.5
87.1
73.2
81.4
85.4
65.0
77.8
GPT-5.4
90.4
40.6
75.3
81.5
56.5
72.4
77.0
49.6
68.2
83.8
48.1
69.5
Gemini 3.1 FL
87.4
50.2
77.3
77.7
68.6
73.9
73.5
64.8
70.9
63.3
47.3
56.2
GPT-5 mini
87.0
51.7
78.5
83.1
66.6
76.5
78.5
63.2
73.4
48.9
19.0
28.2
GPT-5.4 mini
85.8
51.0
77.9
79.1
69.0
73.8
74.7
64.0
70.6
77.1
60.2
68.9
Qwen3-VL-235B
78.8
39.7
70.1
77.0
66.1
73.2
54.4
41.1
48.4
50.5
29.3
38.1
UI-TARS†
—
—
—
—
—
—
—
—
—
12.6
9.0
10.5
Table 3: Pass rate (%) by family and mode, averaged across all six models (excluding UI-TARS). Top: five hardest families; bottom: three easiest. Full breakdown in Figure 3 (Appendix).
Family
BU
AX
SoM
Pix
Avg
Drag/Drop & Workspace
29.7
72.1
31.8
57.4
47.7
Continuous Precision
74.0
53.8
53.9
56.2
59.4
Advanced Editors
83.6
67.8
49.7
46.4
61.9
Date & Time
83.4
76.2
65.8
59.4
71.2
Disclosure & Progressive
80.1
76.0
64.1
68.3
72.1
Discrete Choice
97.2
85.7
83.3
79.0
86.3
Overlays & Transient UI
96.4
91.0
88.6
80.7
89.2
Command & Navigation
97.3
96.1
89.9
83.1
91.6
Table 4: Failure taxonomy over all failed BrowserGym-mode traces, under two complementary labelings: a deterministic trace-feature pass over five models, and Layer-2 LLM diagnostic labels for Gemini 3.1 Flash-Lite mapped onto the same categories. The two labelings surface overlapping high-frequency mechanism families, though their percentages are not directly comparable (different model sets and schemas). Continuous-calibration and drag-execution failures are not represented as separate categories in the original Layer-2 schema, hence absent from that column.
Failure category
Deterministic
Layer-2
(5 models, n=8,864)
(1 model, n=2,752)
Continuous calibration error
20.2%
—
Transient state loss
19.9%
15.0%
Missing commit or confirmation
11.6%
20.0%
Target acquisition / wrong instance
11.2%
31.6%
Repetition / no-progress loop
11.2%
5.6%
Widget-specific procedure missing
9.6%
24.1%
Drag execution failure
4.3%
—
Semantic value error
3.6%
3.3%
Other / unclear
8.4%
0.3%
Table 5: ComponentBench-Core results (% of 912 tasks). Same metrics (and Browser-Use turn-level caveat) as Table 2, on the hard-only Core suite. Pass rates drop 10–39% from Full, confirming that Core concentrates diagnostic mass on unresolved interaction families. Opus 4.6 is evaluated only on Core Pixel.
Model
Mode
Pass
≤𝑯
≤𝟐𝑯
≤𝟑𝑯
Gemini 3 Flash
Browser-Use
84.5
51.5
71.5
78.5
Gemini 3 Flash
Pixel
60.9
30.5
51.0
56.0
GPT-5.4 mini
Browser-Use
57.8
36.8
51.2
55.3
GPT-5.4 mini
Pixel
37.7
22.0
32.1
34.1
Opus 4.6
Pixel
65.4
34.1
53.8
59.4
Table 6: Realized diversity of the 2,910 ComponentBench-Full tasks over libraries, templates, difficulty, and the eight controlled scene factors. The 30 external tasks are the markdown-editor tasks, which use the third-party @uiw/react-md-editor because none of the three primary libraries ships a core markdown editor. One additional ad-hoc template variant (replace_code) is used by a single OTP-input task. The realized difficulty buckets deviate slightly from the prompted 970/970/970 split because two later-added types (breadcrumb, pagination) were generated by a run that did not enforce the per-type balance quota.
Dimension
Levels (count)
Coverage
Canonical type
97 types × 30 tasks
100%
Family
14 families (120–300 tasks each)
100%
Library
antd 1,000 / mui 910 / mantine 970 / external 30
4
Task template
24 templates; most-used match_reference (351),
24/24
least-used file_manage (10)
Difficulty bucket
easy 972 / mid 985 / hard 953
3/3
Difficulty tier
L0 789 / L1 1,050 / L2 818 / L3 253
4/4
Theme
light 2,600 / dark 310
2/2
Spacing
comfortable 2,595 / compact 315
2/2
Scale
default 2,634 / small 272 / large 4
3/3
Clutter
none 1,751 / low 666 / medium 339 / high 154
4/4
Instances
1: 2,266 / 2: 398 / 3: 206 / ≥4: 40
full
Layout
8 layouts; isolated_card 1,866, form_section 244,
8/8
dashboard 210, settings_panel 207, others 383
Placement
center 2,456 / off-center (4 corners) 454
5/5
Guidance
text 2,476 / visual 250 / mixed 184
3/3
Table 7: Full step efficiency on ComponentBench-Full (% of 2,910 tasks). Pass: solved within 20 steps. ≤H/≤2H/≤3H/≤5H: solved within 1/2/3/5× the human step count. Bold: best per model. Browser-Use steps are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. †Native pixel-only interface.
Model
Mode
Pass
≤𝑯
≤𝟐𝑯
≤𝟑𝑯
≤𝟓𝑯
Gemini 3 Flash
Browser-Use
95.2
56.1
85.3
89.1
91.9
AX-tree
89.6
74.7
82.5
84.7
87.4
SoM
87.1
73.2
81.4
83.6
85.2
Pixel
85.4
65.0
77.8
80.9
83.6
GPT-5.4
Browser-Use
90.4
40.6
75.3
82.9
86.8
AX-tree
81.5
56.5
72.4
76.3
79.4
SoM
77.0
49.6
68.2
72.9
75.5
Pixel
83.8
48.1
69.5
75.7
80.0
Gemini 3.1 Flash-Lite
Browser-Use
87.4
50.2
77.3
81.1
83.8
AX-tree
77.7
68.6
73.9
75.3
76.5
SoM
73.5
64.8
70.9
72.2
72.8
Pixel
63.3
47.3
56.2
59.0
61.0
GPT-5 mini
Browser-Use
87.0
51.7
78.5
82.1
84.8
AX-tree
83.1
66.6
76.5
78.8
80.6
SoM
78.5
63.2
73.4
75.9
77.3
Pixel
48.9
19.0
28.2
32.7
38.7
GPT-5.4 mini
Browser-Use
85.8
51.0
77.9
81.6
83.7
AX-tree
79.1
69.0
73.8
75.4
76.9
SoM
74.7
64.0
70.6
72.2
73.5
Pixel
77.1
60.2
68.9
72.1
74.9
Qwen3-VL-235B
Browser-Use
78.8
39.7
70.1
74.2
77.0
AX-tree
77.0
66.1
73.2
74.8
75.7
SoM
54.4
41.1
48.4
51.3
53.0
Pixel
50.5
29.3
38.1
42.3
46.3
UI-TARS-1.5-7B†
Pixel
12.6
9.0
10.5
11.2
11.9
Table 8: Full time efficiency on successful ComponentBench-Full tasks. Human and Agent columns report mean wall-clock duration over the same task set: the tasks each model–mode configuration solved (hence the Human column varies by row). Ratio is agent-to-human mean time.
Model
Mode
Human
Agent
Ratio
Med. Agent
Gemini 3 Flash
AX-tree
4.5s
21.5s
4.7×
13.5s
SoM
4.6s
22.0s
4.8×
14.4s
Pixel
4.3s
27.6s
6.4×
17.1s
Browser-Use
4.6s
32.6s
7.1×
21.2s
Gemini 3.1 Flash-Lite
AX-tree
4.1s
14.5s
3.6×
11.2s
SoM
3.8s
14.4s
3.8×
11.6s
Pixel
3.1s
15.8s
5.1×
11.8s
Browser-Use
4.5s
23.1s
5.1×
17.7s
GPT-5.4
AX-tree
4.0s
31.9s
7.9×
19.2s
SoM
3.7s
35.0s
9.4×
22.3s
Pixel
4.2s
23.0s
5.5×
14.2s
Browser-Use
4.6s
36.2s
7.8×
24.3s
GPT-5 mini
AX-tree
4.5s
32.1s
7.2×
18.9s
SoM
4.3s
32.7s
7.7×
19.9s
Pixel
3.3s
71.8s
21.5×
41.4s
Browser-Use
4.6s
36.3s
8.0×
24.2s
GPT-5.4 mini
AX-tree
3.9s
14.7s
3.8×
11.4s
SoM
3.9s
14.2s
3.7×
11.4s
Pixel
3.6s
14.2s
3.9×
10.7s
Browser-Use
4.2s
17.6s
4.2×
13.7s
Qwen3-VL-235B
AX-tree
3.9s
21.6s
5.6×
13.8s
SoM
3.4s
24.4s
7.1×
14.4s
Pixel
2.9s
26.8s
9.3×
16.0s
Browser-Use
4.1s
45.0s
10.9×
28.1s
UI-TARS-1.5-7B
Native Pixel
2.4s
15.5s
6.5×
10.1s
Table 9: SoM–Pixel delta on ComponentBench-Full (%). The SoM advantage is model-dependent, ranging from +29.6% to −6.8%.
Model
SoM
Pixel
Δ (SoM−Pixel)
GPT-5 mini
78.5
48.9
+29.6
Gemini 3.1 Flash-Lite
73.5
63.3
+10.2
Qwen3-VL-235B
54.4
50.5
+3.9
Gemini 3 Flash
87.1
85.4
+1.7
GPT-5.4 mini
74.7
77.1
−2.4
GPT-5.4
77.0
83.8
−6.8
Table 10: Browser-Use advantage over mean non-Browser-Use pass rate (%). The advantage is largest for weaker models.
Model
Browser-Use
Mean(AX,SoM,Pix)
Δ
Qwen3-VL-235B
78.8
60.6
+18.2
GPT-5 mini
87.0
70.2
+16.8
Gemini 3.1 Flash-Lite
87.4
71.5
+15.9
GPT-5.4
90.4
80.8
+9.6
GPT-5.4 mini
85.8
77.0
+8.8
Gemini 3 Flash
95.2
87.4
+7.8
Table 11: Pearson correlation between intended difficulty axes and agent failure rate. Precision requirement is the most predictive axis overall; all axes predict Pixel failure more strongly than Browser-Use failure.
Difficulty axis
Overall
AX-tree
Pixel
Browser-Use
Precision requirement
+0.44
+0.30
+0.41
+0.33
Target acquisition
+0.31
+0.17
+0.36
+0.19
Density / choice interf.
+0.23
+0.10
+0.31
+0.12
Feedback dynamics
+0.23
+0.12
+0.29
+0.13
Depth / layering
+0.20
+0.11
+0.29
+0.05
Semantic observability
+0.13
+0.08
+0.17
+0.05
Disambiguation load
+0.09
+0.04
+0.15
+0.02
Table 12: Pass rate by intended difficulty tier and observation mode, averaged across all six models (excluding UI-TARS). The AX-tree–Pixel gap widens monotonically from 4.6% (L0) to 21.9% (L3).
Tier
AX-tree
SoM
Pixel
Browser-Use
L0 (easy)
87.7%
85.9%
83.0%
92.8%
L1 (medium)
82.0%
75.1%
69.8%
87.6%
L2 (hard)
77.4%
66.4%
57.6%
84.3%
L3 (hard+)
71.4%
59.4%
49.5%
80.4%
Table 13: The 15 hardest canonical types by mean agent pass rate (averaged across all models and modes, excluding UI-TARS). Human steps column shows the mean normalized human reference steps. Components with ≤2 human steps but <60% agent pass rate represent the human-agent difficulty inversion.
Component type
Agent %
Human steps
Family
resizable_columns
24.4
1.7
Drag/Drop
window_splitter
38.3
1.3
Disclosure
slider_range
39.9
1.9
Continuous
rich_text_editor
40.7
4.9
Adv. Editors
meter
46.5
1.6
Continuous
datetime_picker_range
48.9
10.5
Date/Time
alpha_slider
50.0
1.5
Continuous
kanban_board_drag_drop
52.1
1.4
Drag/Drop
select_native
53.1
1.4
List Selection
feed_infinite_scroll
53.3
4.3
Disclosure
color_picker_2d
55.7
2.9
Continuous
virtual_list
56.7
3.3
Structured Data
drag_drop_between_lists
56.9
1.7
Drag/Drop
drag_drop_sortable_list
57.5
1.8
Drag/Drop
code_editor
59.0
3.9
Adv. Editors
Table 14: Agreement between the original reference annotator and two additional annotators (A1, A2) on the 278-task validation subset (action tasks only). Human-to-human variation is small at the aggregate level.
Metric
Value
Mean normalized steps (Original / A1 / A2)
2.92 / 2.90 / 3.14
Median normalized steps (all annotators)
2
Pairwise step-count Pearson correlation
0.79–0.94
Exact step-count agreement
71%–84%
Mean absolute step-count difference
0.32–0.64
Table 15: Repeated-run stability on the 278-task subset (2 runs per cell, same harness commit and endpoint). Max dev: run-to-run pass-rate deviation (%). Agree%: task-level exact pass/fail agreement. Jaccard: overlap of passed-task sets. CI: bootstrap 95% interval on the mean pass rate.
Model
Mode
Mean pass (%)
Max dev
Agree%
Jaccard
95% CI
Gemini 3 Flash
Browser-Use
95.1
1.1
97.5
0.974
[92.6, 97.3]
Gemini 3 Flash
Pixel
87.4
0.7
90.6
0.898
[84.0, 90.8]
Gemini 3.1 Flash-Lite
Browser-Use
87.1
0.7
94.2
0.936
[83.3, 90.6]
Gemini 3.1 Flash-Lite
Pixel
77.0
0.0
86.3
0.837
[72.5, 81.3]
GPT-5 mini
Browser-Use
86.7
1.4
92.1
0.913
[82.9, 90.3]
GPT-5 mini
Pixel
52.5
0.0
82.7
0.718
[47.1, 57.9]
GPT-5.4 mini
Browser-Use
85.4
1.1
91.0
0.900
[81.5, 89.0]
GPT-5.4 mini
Pixel
80.6
1.4
87.8
0.859
[76.3, 84.7]
왜 중요한가
실제 서비스에 배치되는 컴퓨터 조작 AI 에이전트가 어떤 화면 요소에서, 왜 실패하는지 구체적으로 짚어낼 수 있게 해준다. 전체 작업 흐름 평가만으로는 보이지 않던 특정 부품 단위의 취약점을 드러내, 에이전트 설계자가 어디를 고쳐야 할지 판단하는 데 도움을 준다.
이 논문의 용어
AX-tree(접근성 트리) · 화면 요소의 이름, 역할 등을 텍스트로 정리한 정보로, 스크린리더 등이 활용하는 구조
Set-of-Marks(SoM) · 스크린샷 위에 클릭 가능한 요소마다 번호를 매겨 표시해주는 방식
Browser-Use · 스크린샷 외에 웹페이지의 DOM(문서 구조) 정보까지 제공해 한 번에 여러 동작을 수행할 수 있게 하는 도구 기반 평가 방식
프로그래밍 방식 검증기(programmatic verifier) · 작업이 실제로 끝난 상태인지 코드로 자동 확인하는 장치
본문에 싣지 못한 그림
Figure 1: ComponentBench evaluates computer-use agents on 97 canonical UI component types organized into 14 families. Left: Tasks span diverse interaction types implemented across Ant Design, MUI, and Mantine. Right: The ontology covers the breadth of modern web UI interaction.
Figure 2: A concrete ComponentBench task (data_table_filterable-mantine-T10). (a) The agent sees a rendered page containing three visually similar mini-tables. (b) The instruction, programmatic verifier, and structured metadata. The agent must disambiguate the correct table instance (Invoices), apply two filters, and commit via the instance-local Apply button.
Figure 3: Pass rate (%) by component family, model, and observation mode on ComponentBench-Full. Families sorted by difficulty (hardest at top). The heatmap reveals where mode ordering inverts: Drag/Drop favors Pixel over Browser-Use; Advanced Editors strongly favor Browser-Use.
Figure 4: Human-agent difficulty inversion. Each point is one canonical type. The lower-left quadrant contains components trivial for humans (≤2 steps) but hard for agents (<60% pass).
Figure 5: Clutter and spacing disproportionately hurt Pixel mode, widening the AX-tree–Pixel gap from 10% to 24% (clutter) and 12% to 21% (spacing).
Figure 6: GPT-5.4 mini SoM vs. Pixel task outcomes by component family. For each family, tasks are classified as both-pass, SoM-only, Pixel-only, or both-fail. Drag/Drop and Continuous Precision families show the strongest Pixel-over-SoM advantage.
Figure 7: Browser-Use (CUA) advantage by component family, averaged across the six models evaluated in all four regimes. The advantage ranges from +29% (Advanced Editors) to −24% (Drag/Drop), confirming that DOM-level tool access is not uniformly beneficial.
Figure 8: Pass rate by task template and observation mode. Templates requiring spatial control (set_range, drag_operation) or complex editing (editor_operation) are substantially harder than simple activation or disclosure tasks.
Figure 9: Efficiency frontier: pass rate vs. mean time per successful task. Each point is one model-mode combination among the six models evaluated in the shared and Browser-Use regimes; the native UI-TARS configuration is omitted. The human reference point (100% pass, 4.8s) is shown for comparison.
Figure 10: Marginal tasks solved per additional step (Gemini 3 Flash). Most diagnostic value is concentrated in the first 5 steps; steps 6–20 contribute diminishing returns.
