Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents
arXiv:2608.183072026-08-20
AI agents that click and drag on real screens still stumble on small UI parts like sliders and drag-and-drop lists
Researchers built ComponentBench, a benchmark that tests computer-use AI agents on individual web UI components rather than whole workflows. It packages 97 canonical component types like date pickers, sliders, and drag-and-drop lists into 2,910 working tasks built with real UI libraries, each checked against a cleaned human reference recording. Across seven models and four ways of presenting the screen, success rates swung by more than 30 percentage points for the same model, and agents were still far slower and weaker than humans on simple spatial actions like dragging or resizing.
METAL MEDIA explanatory visual
AI agents that click and drag on real screens still stumble on small UI parts like sliders and drag-and-drop lists
01ComponentBench evaluates agents on isolated UI components (e.g., a filterable data table or a set of toggle buttons) rather than long multi-step workflows, using 2,910 tasks built with Ant Design, MUI, and Mantine component libraries and checked by automatic verifiers.
02Seven models -- including GPT-5.4, Gemini 3 Flash, GPT-5 mini, and Qwen3-VL-235B -- were tested under four observation/action setups: accessibility-tree text, numbered overlay marks, screenshot-only pixel coordinates, and a DOM-aware tool called Browser-Use. GPT-5 mini's pass rate dropped from 83.1% with accessibility-tree text to 48.9% with pixel-only coordinates, a swing of over 30 points from interface choice alone.
03Even the fastest model-mode combination took 3.7 times as long as the matched human reference trace to finish tasks, and several spatial tasks that humans finish in one or two steps -- resizable columns, window splitters, range sliders -- stayed below 60% pass rate for every agent tested.
04Screenshot-only agents were disproportionately hurt by visually cluttered or tightly spaced interfaces, while DOM-aware Browser-Use was largely unaffected by clutter, showing the failure is about visual grounding rather than understanding the task.
05A trace-by-trace failure analysis found agents often locate the right control but can't calibrate a continuous value (like dragging a slider precisely) or lose an open menu/edit mode before confirming -- mechanism-level failures rather than misunderstanding instructions.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
ComponentBench evaluates agents on isolated UI components (e.g., a filterable data table or a set of toggle buttons) rather than long multi-step workflows, using 2,910 tasks built with Ant Design, MUI, and Mantine component libraries and checked by automatic verifiers.
Seven models -- including GPT-5.4, Gemini 3 Flash, GPT-5 mini, and Qwen3-VL-235B -- were tested under four observation/action setups: accessibility-tree text, numbered overlay marks, screenshot-only pixel coordinates, and a DOM-aware tool called Browser-Use. GPT-5 mini's pass rate dropped from 83.1% with accessibility-tree text to 48.9% with pixel-only coordinates, a swing of over 30 points from interface choice alone.
Even the fastest model-mode combination took 3.7 times as long as the matched human reference trace to finish tasks, and several spatial tasks that humans finish in one or two steps -- resizable columns, window splitters, range sliders -- stayed below 60% pass rate for every agent tested.
Screenshot-only agents were disproportionately hurt by visually cluttered or tightly spaced interfaces, while DOM-aware Browser-Use was largely unaffected by clutter, showing the failure is about visual grounding rather than understanding the task.
A trace-by-trace failure analysis found agents often locate the right control but can't calibrate a continuous value (like dragging a slider precisely) or lose an open menu/edit mode before confirming -- mechanism-level failures rather than misunderstanding instructions.
Table 1: ComponentBench-Full pass rates (%) by model and observation/action space. All runs use a 20-step budget. Browser-Use is a tool-rich reference regime with DOM access; AX-tree, SoM, and Pixel share the same harness and differ only in observation and action space. Task-resampling bootstrap 95% CIs (B=10,000) have half-width ≤1.9% per cell. †Native pixel-only harness with coordinate prediction. ‡Browser-Use served via Amazon Bedrock (precision unspecified); other Qwen cells use local vLLM FP8 (Appendix M).
Model
Browser-Use
AX-tree
SoM
Pixel
Gemini 3 Flash
95.2
89.6
87.1
85.4
GPT-5.4
90.4
81.5
77.0
83.8
Gemini 3.1 Flash-Lite
87.4
77.7
73.5
63.3
GPT-5 mini
87.0
83.1
78.5
48.9
GPT-5.4 mini
85.8
79.1
74.7
77.1
Qwen3-VL-235B‡
78.8
77.0
54.4
50.5
UI-TARS-1.5-7B†
—
—
—
12.6
Table 2: Step efficiency (% of 2,910 tasks). ≤H/≤2H: solved within 1/2× human steps, where H=max(human steps,1); hover-only references count as H=1. Browser-Use “steps” are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. †Native pixel-only harness. Full ≤3H/5H breakdown in Appendix C.
Browser-Use
AX-tree
SoM
Pixel
Model
Pass
≤H
≤2H
Pass
≤H
≤2H
Pass
≤H
≤2H
Pass
≤H
≤2H
Gemini 3 Flash
95.2
56.1
85.3
89.6
74.7
82.5
87.1
73.2
81.4
85.4
65.0
77.8
GPT-5.4
90.4
40.6
75.3
81.5
56.5
72.4
77.0
49.6
68.2
83.8
48.1
69.5
Gemini 3.1 FL
87.4
50.2
77.3
77.7
68.6
73.9
73.5
64.8
70.9
63.3
47.3
56.2
GPT-5 mini
87.0
51.7
78.5
83.1
66.6
76.5
78.5
63.2
73.4
48.9
19.0
28.2
GPT-5.4 mini
85.8
51.0
77.9
79.1
69.0
73.8
74.7
64.0
70.6
77.1
60.2
68.9
Qwen3-VL-235B
78.8
39.7
70.1
77.0
66.1
73.2
54.4
41.1
48.4
50.5
29.3
38.1
UI-TARS†
—
—
—
—
—
—
—
—
—
12.6
9.0
10.5
Table 3: Pass rate (%) by family and mode, averaged across all six models (excluding UI-TARS). Top: five hardest families; bottom: three easiest. Full breakdown in Figure 3 (Appendix).
Family
BU
AX
SoM
Pix
Avg
Drag/Drop & Workspace
29.7
72.1
31.8
57.4
47.7
Continuous Precision
74.0
53.8
53.9
56.2
59.4
Advanced Editors
83.6
67.8
49.7
46.4
61.9
Date & Time
83.4
76.2
65.8
59.4
71.2
Disclosure & Progressive
80.1
76.0
64.1
68.3
72.1
Discrete Choice
97.2
85.7
83.3
79.0
86.3
Overlays & Transient UI
96.4
91.0
88.6
80.7
89.2
Command & Navigation
97.3
96.1
89.9
83.1
91.6
Table 4: Failure taxonomy over all failed BrowserGym-mode traces, under two complementary labelings: a deterministic trace-feature pass over five models, and Layer-2 LLM diagnostic labels for Gemini 3.1 Flash-Lite mapped onto the same categories. The two labelings surface overlapping high-frequency mechanism families, though their percentages are not directly comparable (different model sets and schemas). Continuous-calibration and drag-execution failures are not represented as separate categories in the original Layer-2 schema, hence absent from that column.
Failure category
Deterministic
Layer-2
(5 models, n=8,864)
(1 model, n=2,752)
Continuous calibration error
20.2%
—
Transient state loss
19.9%
15.0%
Missing commit or confirmation
11.6%
20.0%
Target acquisition / wrong instance
11.2%
31.6%
Repetition / no-progress loop
11.2%
5.6%
Widget-specific procedure missing
9.6%
24.1%
Drag execution failure
4.3%
—
Semantic value error
3.6%
3.3%
Other / unclear
8.4%
0.3%
Table 5: ComponentBench-Core results (% of 912 tasks). Same metrics (and Browser-Use turn-level caveat) as Table 2, on the hard-only Core suite. Pass rates drop 10–39% from Full, confirming that Core concentrates diagnostic mass on unresolved interaction families. Opus 4.6 is evaluated only on Core Pixel.
Model
Mode
Pass
≤𝑯
≤𝟐𝑯
≤𝟑𝑯
Gemini 3 Flash
Browser-Use
84.5
51.5
71.5
78.5
Gemini 3 Flash
Pixel
60.9
30.5
51.0
56.0
GPT-5.4 mini
Browser-Use
57.8
36.8
51.2
55.3
GPT-5.4 mini
Pixel
37.7
22.0
32.1
34.1
Opus 4.6
Pixel
65.4
34.1
53.8
59.4
Table 6: Realized diversity of the 2,910 ComponentBench-Full tasks over libraries, templates, difficulty, and the eight controlled scene factors. The 30 external tasks are the markdown-editor tasks, which use the third-party @uiw/react-md-editor because none of the three primary libraries ships a core markdown editor. One additional ad-hoc template variant (replace_code) is used by a single OTP-input task. The realized difficulty buckets deviate slightly from the prompted 970/970/970 split because two later-added types (breadcrumb, pagination) were generated by a run that did not enforce the per-type balance quota.
Dimension
Levels (count)
Coverage
Canonical type
97 types × 30 tasks
100%
Family
14 families (120–300 tasks each)
100%
Library
antd 1,000 / mui 910 / mantine 970 / external 30
4
Task template
24 templates; most-used match_reference (351),
24/24
least-used file_manage (10)
Difficulty bucket
easy 972 / mid 985 / hard 953
3/3
Difficulty tier
L0 789 / L1 1,050 / L2 818 / L3 253
4/4
Theme
light 2,600 / dark 310
2/2
Spacing
comfortable 2,595 / compact 315
2/2
Scale
default 2,634 / small 272 / large 4
3/3
Clutter
none 1,751 / low 666 / medium 339 / high 154
4/4
Instances
1: 2,266 / 2: 398 / 3: 206 / ≥4: 40
full
Layout
8 layouts; isolated_card 1,866, form_section 244,
8/8
dashboard 210, settings_panel 207, others 383
Placement
center 2,456 / off-center (4 corners) 454
5/5
Guidance
text 2,476 / visual 250 / mixed 184
3/3
Table 7: Full step efficiency on ComponentBench-Full (% of 2,910 tasks). Pass: solved within 20 steps. ≤H/≤2H/≤3H/≤5H: solved within 1/2/3/5× the human step count. Bold: best per model. Browser-Use steps are model turns of up to 4 primitive actions (page-load turn excluded), so its efficiency columns are optimistic. †Native pixel-only interface.
Model
Mode
Pass
≤𝑯
≤𝟐𝑯
≤𝟑𝑯
≤𝟓𝑯
Gemini 3 Flash
Browser-Use
95.2
56.1
85.3
89.1
91.9
AX-tree
89.6
74.7
82.5
84.7
87.4
SoM
87.1
73.2
81.4
83.6
85.2
Pixel
85.4
65.0
77.8
80.9
83.6
GPT-5.4
Browser-Use
90.4
40.6
75.3
82.9
86.8
AX-tree
81.5
56.5
72.4
76.3
79.4
SoM
77.0
49.6
68.2
72.9
75.5
Pixel
83.8
48.1
69.5
75.7
80.0
Gemini 3.1 Flash-Lite
Browser-Use
87.4
50.2
77.3
81.1
83.8
AX-tree
77.7
68.6
73.9
75.3
76.5
SoM
73.5
64.8
70.9
72.2
72.8
Pixel
63.3
47.3
56.2
59.0
61.0
GPT-5 mini
Browser-Use
87.0
51.7
78.5
82.1
84.8
AX-tree
83.1
66.6
76.5
78.8
80.6
SoM
78.5
63.2
73.4
75.9
77.3
Pixel
48.9
19.0
28.2
32.7
38.7
GPT-5.4 mini
Browser-Use
85.8
51.0
77.9
81.6
83.7
AX-tree
79.1
69.0
73.8
75.4
76.9
SoM
74.7
64.0
70.6
72.2
73.5
Pixel
77.1
60.2
68.9
72.1
74.9
Qwen3-VL-235B
Browser-Use
78.8
39.7
70.1
74.2
77.0
AX-tree
77.0
66.1
73.2
74.8
75.7
SoM
54.4
41.1
48.4
51.3
53.0
Pixel
50.5
29.3
38.1
42.3
46.3
UI-TARS-1.5-7B†
Pixel
12.6
9.0
10.5
11.2
11.9
Table 8: Full time efficiency on successful ComponentBench-Full tasks. Human and Agent columns report mean wall-clock duration over the same task set: the tasks each model–mode configuration solved (hence the Human column varies by row). Ratio is agent-to-human mean time.
Model
Mode
Human
Agent
Ratio
Med. Agent
Gemini 3 Flash
AX-tree
4.5s
21.5s
4.7×
13.5s
SoM
4.6s
22.0s
4.8×
14.4s
Pixel
4.3s
27.6s
6.4×
17.1s
Browser-Use
4.6s
32.6s
7.1×
21.2s
Gemini 3.1 Flash-Lite
AX-tree
4.1s
14.5s
3.6×
11.2s
SoM
3.8s
14.4s
3.8×
11.6s
Pixel
3.1s
15.8s
5.1×
11.8s
Browser-Use
4.5s
23.1s
5.1×
17.7s
GPT-5.4
AX-tree
4.0s
31.9s
7.9×
19.2s
SoM
3.7s
35.0s
9.4×
22.3s
Pixel
4.2s
23.0s
5.5×
14.2s
Browser-Use
4.6s
36.2s
7.8×
24.3s
GPT-5 mini
AX-tree
4.5s
32.1s
7.2×
18.9s
SoM
4.3s
32.7s
7.7×
19.9s
Pixel
3.3s
71.8s
21.5×
41.4s
Browser-Use
4.6s
36.3s
8.0×
24.2s
GPT-5.4 mini
AX-tree
3.9s
14.7s
3.8×
11.4s
SoM
3.9s
14.2s
3.7×
11.4s
Pixel
3.6s
14.2s
3.9×
10.7s
Browser-Use
4.2s
17.6s
4.2×
13.7s
Qwen3-VL-235B
AX-tree
3.9s
21.6s
5.6×
13.8s
SoM
3.4s
24.4s
7.1×
14.4s
Pixel
2.9s
26.8s
9.3×
16.0s
Browser-Use
4.1s
45.0s
10.9×
28.1s
UI-TARS-1.5-7B
Native Pixel
2.4s
15.5s
6.5×
10.1s
Table 9: SoM–Pixel delta on ComponentBench-Full (%). The SoM advantage is model-dependent, ranging from +29.6% to −6.8%.
Model
SoM
Pixel
Δ (SoM−Pixel)
GPT-5 mini
78.5
48.9
+29.6
Gemini 3.1 Flash-Lite
73.5
63.3
+10.2
Qwen3-VL-235B
54.4
50.5
+3.9
Gemini 3 Flash
87.1
85.4
+1.7
GPT-5.4 mini
74.7
77.1
−2.4
GPT-5.4
77.0
83.8
−6.8
Table 10: Browser-Use advantage over mean non-Browser-Use pass rate (%). The advantage is largest for weaker models.
Model
Browser-Use
Mean(AX,SoM,Pix)
Δ
Qwen3-VL-235B
78.8
60.6
+18.2
GPT-5 mini
87.0
70.2
+16.8
Gemini 3.1 Flash-Lite
87.4
71.5
+15.9
GPT-5.4
90.4
80.8
+9.6
GPT-5.4 mini
85.8
77.0
+8.8
Gemini 3 Flash
95.2
87.4
+7.8
Table 11: Pearson correlation between intended difficulty axes and agent failure rate. Precision requirement is the most predictive axis overall; all axes predict Pixel failure more strongly than Browser-Use failure.
Difficulty axis
Overall
AX-tree
Pixel
Browser-Use
Precision requirement
+0.44
+0.30
+0.41
+0.33
Target acquisition
+0.31
+0.17
+0.36
+0.19
Density / choice interf.
+0.23
+0.10
+0.31
+0.12
Feedback dynamics
+0.23
+0.12
+0.29
+0.13
Depth / layering
+0.20
+0.11
+0.29
+0.05
Semantic observability
+0.13
+0.08
+0.17
+0.05
Disambiguation load
+0.09
+0.04
+0.15
+0.02
Table 12: Pass rate by intended difficulty tier and observation mode, averaged across all six models (excluding UI-TARS). The AX-tree–Pixel gap widens monotonically from 4.6% (L0) to 21.9% (L3).
Tier
AX-tree
SoM
Pixel
Browser-Use
L0 (easy)
87.7%
85.9%
83.0%
92.8%
L1 (medium)
82.0%
75.1%
69.8%
87.6%
L2 (hard)
77.4%
66.4%
57.6%
84.3%
L3 (hard+)
71.4%
59.4%
49.5%
80.4%
Table 13: The 15 hardest canonical types by mean agent pass rate (averaged across all models and modes, excluding UI-TARS). Human steps column shows the mean normalized human reference steps. Components with ≤2 human steps but <60% agent pass rate represent the human-agent difficulty inversion.
Component type
Agent %
Human steps
Family
resizable_columns
24.4
1.7
Drag/Drop
window_splitter
38.3
1.3
Disclosure
slider_range
39.9
1.9
Continuous
rich_text_editor
40.7
4.9
Adv. Editors
meter
46.5
1.6
Continuous
datetime_picker_range
48.9
10.5
Date/Time
alpha_slider
50.0
1.5
Continuous
kanban_board_drag_drop
52.1
1.4
Drag/Drop
select_native
53.1
1.4
List Selection
feed_infinite_scroll
53.3
4.3
Disclosure
color_picker_2d
55.7
2.9
Continuous
virtual_list
56.7
3.3
Structured Data
drag_drop_between_lists
56.9
1.7
Drag/Drop
drag_drop_sortable_list
57.5
1.8
Drag/Drop
code_editor
59.0
3.9
Adv. Editors
Table 14: Agreement between the original reference annotator and two additional annotators (A1, A2) on the 278-task validation subset (action tasks only). Human-to-human variation is small at the aggregate level.
Metric
Value
Mean normalized steps (Original / A1 / A2)
2.92 / 2.90 / 3.14
Median normalized steps (all annotators)
2
Pairwise step-count Pearson correlation
0.79–0.94
Exact step-count agreement
71%–84%
Mean absolute step-count difference
0.32–0.64
Table 15: Repeated-run stability on the 278-task subset (2 runs per cell, same harness commit and endpoint). Max dev: run-to-run pass-rate deviation (%). Agree%: task-level exact pass/fail agreement. Jaccard: overlap of passed-task sets. CI: bootstrap 95% interval on the mean pass rate.
Model
Mode
Mean pass (%)
Max dev
Agree%
Jaccard
95% CI
Gemini 3 Flash
Browser-Use
95.1
1.1
97.5
0.974
[92.6, 97.3]
Gemini 3 Flash
Pixel
87.4
0.7
90.6
0.898
[84.0, 90.8]
Gemini 3.1 Flash-Lite
Browser-Use
87.1
0.7
94.2
0.936
[83.3, 90.6]
Gemini 3.1 Flash-Lite
Pixel
77.0
0.0
86.3
0.837
[72.5, 81.3]
GPT-5 mini
Browser-Use
86.7
1.4
92.1
0.913
[82.9, 90.3]
GPT-5 mini
Pixel
52.5
0.0
82.7
0.718
[47.1, 57.9]
GPT-5.4 mini
Browser-Use
85.4
1.1
91.0
0.900
[81.5, 89.0]
GPT-5.4 mini
Pixel
80.6
1.4
87.8
0.859
[76.3, 84.7]
Why it matters
This gives builders of computer-use agents a way to pinpoint exactly which interface elements and interaction styles cause failures, something invisible in end-to-end task scores. It matters for anyone deploying AI agents to operate real software, since it shows small, seemingly trivial UI interactions remain a major reliability bottleneck.
Terms in this paper
AX-tree (accessibility tree) · a text description of on-screen elements' names and roles, normally used by screen readers
Set-of-Marks (SoM) · a screenshot overlaid with numbered labels on clickable elements
Browser-Use · a tool-based evaluation mode that gives the agent both a screenshot and the page's underlying DOM structure, letting it act via richer controls
programmatic verifier · code that automatically checks whether a task's end state was actually achieved
Figures we cannot republish
Figure 1: ComponentBench evaluates computer-use agents on 97 canonical UI component types organized into 14 families. Left: Tasks span diverse interaction types implemented across Ant Design, MUI, and Mantine. Right: The ontology covers the breadth of modern web UI interaction.
Figure 2: A concrete ComponentBench task (data_table_filterable-mantine-T10). (a) The agent sees a rendered page containing three visually similar mini-tables. (b) The instruction, programmatic verifier, and structured metadata. The agent must disambiguate the correct table instance (Invoices), apply two filters, and commit via the instance-local Apply button.
Figure 3: Pass rate (%) by component family, model, and observation mode on ComponentBench-Full. Families sorted by difficulty (hardest at top). The heatmap reveals where mode ordering inverts: Drag/Drop favors Pixel over Browser-Use; Advanced Editors strongly favor Browser-Use.
Figure 4: Human-agent difficulty inversion. Each point is one canonical type. The lower-left quadrant contains components trivial for humans (≤2 steps) but hard for agents (<60% pass).
Figure 5: Clutter and spacing disproportionately hurt Pixel mode, widening the AX-tree–Pixel gap from 10% to 24% (clutter) and 12% to 21% (spacing).
Figure 6: GPT-5.4 mini SoM vs. Pixel task outcomes by component family. For each family, tasks are classified as both-pass, SoM-only, Pixel-only, or both-fail. Drag/Drop and Continuous Precision families show the strongest Pixel-over-SoM advantage.
Figure 7: Browser-Use (CUA) advantage by component family, averaged across the six models evaluated in all four regimes. The advantage ranges from +29% (Advanced Editors) to −24% (Drag/Drop), confirming that DOM-level tool access is not uniformly beneficial.
Figure 8: Pass rate by task template and observation mode. Templates requiring spatial control (set_range, drag_operation) or complex editing (editor_operation) are substantially harder than simple activation or disclosure tasks.
Figure 9: Efficiency frontier: pass rate vs. mean time per successful task. Each point is one model-mode combination among the six models evaluated in the shared and Browser-Use regimes; the native UI-TARS configuration is omitted. The human reference point (100% pass, 4.8s) is shown for comparison.
Figure 10: Marginal tasks solved per additional step (Gemini 3 Flash). Most diagnostic value is concentrated in the first 5 steps; steps 6–20 contribute diminishing returns.
Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.
