Figure 1: Example agent workflow task in DSAgentBench, showing selected stages from a multi-step workflow: the agent retrieves data, executes code, and produces the final outputs. The illustrated trajectory is generated by GPT-4o.
Table 1: Comparison of DSAgentBench with existing benchmarks across unified agent and data-science capabilities. ✓ = Yes, ▲ = Partial, ✗ = No.
Benchmark
Full OS Interaction
Web Tools & Browsers
Terminal + GUI Control
Cross-App Usage
Intermediate State
Controlled Exec Env
Multimodal Support
Data-Science Tasks
Visualization Evaluation
Environment Scalability
HumanEval chen2021evaluating
✗
✗
✗
✗
✗
✗
✗
✗
✗
✗
KRAMABench lai2025kramabench
✗
✗
✗
✗
✓
code
✗
▲
✗
✗
DS-1000 lai2023ds
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DABStep egg2025dabstep
✗
✗
✗
✗
✓
code
✗
▲
✗
✗
MLAgentBench huang2023mlagentbench
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DSBench jing2024dsbench
✗
✗
✗
✗
✗
✗
✓
▲
✗
✗
DSEval zhang2024benchmarking
✗
✗
✗
✗
✗
✓
✗
▲
✗
✗
ARCADE yin2023natural
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DA-CODE huang2024code
✗
✗
✗
✗
✗
code
✗
▲
▲
▲
OSWORLD xie2024osworld
✓
✓
✓
✓
✓
✓
✓
✗
✗
✓
DSAgentBench (Ours)
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Figure 3: Task category distribution in our benchmark, reflecting real-world data science practice.
Figure 4: DSAgentBench execution workflow: the agent perceives screenshots and the A11y tree, executes GUI-based actions, and is evaluated by a custom script.
Table 3: DSAgentBench accuracy (%) under two observation settings (Screenshot and Screenshot + A11y Tree) across data-science lifecycle tasks: Data Acquisition (DA), Exploratory Data Analysis (EDA), Feature Engineering (FE), Modeling, Visualization (Vis), and Evaluation (Eval). Results for open-source models are reported in Table 13.
Model
Screenshot
Screenshot + A11y Tree
DA
EDA
FE
Model
Vis
Eval
Overall
DA
EDA
FE
Model
Vis
Eval
Overall
GPT-4o
0.00
27.06
24.24
16.67
7.14
33.33
19.34
13.04
29.66
35.14
21.95
15.62
8.33
24.54
O4-mini
4.35
1.68
2.70
0.00
3.03
0.00
1.82
0.00
4.20
5.41
0.00
0.00
0.00
2.55
GPT-5-mini
8.70
17.28
25.00
3.57
21.43
0.00
15.20
4.55
24.14
27.03
7.50
21.88
0.00
19.03
GPT-5
17.39
27.41
27.03
26.83
9.09
16.67
23.63
26.08
33.44
29.72
32.22
15.15
33.33
29.81
Gemini-2.5-Pro
4.55
17.86
16.13
17.24
10.71
16.67
14.49
4.35
22.78
33.33
14.81
30.77
0.00
20.81
OpenAI CUA
0.00
10.59
12.50
0.00
14.29
0.00
8.13
4.35
9.52
6.06
2.86
6.67
0.00
6.61
Claude-4-Sonnet
0.00
7.46
4.17
0.00
3.85
0.00
4.55
0.00
4.86
0.00
1.00
11.11
0.00
4.64
Claude-4.5-Sonnet
4.76
4.17
0.00
3.45
3.23
0.00
3.17
8.33
12.31
12.50
4.55
4.17
0.00
9.21
Claude-4.6-Sonnet
43.48
57.98
51.35
41.46
39.39
50.00
50.55
47.82
64.88
56.75
46.34
42.42
66.67
56.70
Human Performance
73.91
90.76
91.89
73.17
78.79
83.33
85.09
73.91
90.76
91.89
73.17
78.79
83.33
85.09
Figure 5: Example environments in DSAgentBench: agent interaction across data-science environments, demonstrating autonomous coordination of file systems, databases, notebooks, IDEs, visualization, execution, debugging, and web-based data retrieval.
Table 4: DSAgentBench accuracy (%) across stage type, task complexity, and tool usage under different settings.
Model
Screenshot
Screenshot + A11y Tree
Single
Multi
Easy
Medium
Hard
VS Code
Jupyter
Single
Multi
Easy
Medium
Hard
VS Code
Jupyter
GPT-4o
26.97
13.52
38.46
24.74
11.79
14.69
39.22
35.59
16.13
46.67
34.38
12.31
23.87
27.45
O4-mini
4.20
0.00
20.00
1.55
0.00
1.79
1.92
5.04
0.64
13.33
3.88
0.00
2.69
1.92
GPT-5-mini
24.71
8.00
38.46
24.47
9.41
15.23
11.00
31.03
9.87
66.67
27.20
5.47
14.68
38.00
GPT-5
27.73
20.51
53.33
27.13
16.79
25.10
17.30
39.49
22.43
66.67
32.55
22.89
27.35
40.38
Gemini-2.5-Pro
19.12
10.92
46.15
15.62
9.18
17.31
2.40
32.10
12.93
53.85
28.74
9.28
17.45
31.25
OpenAI CUA
13.64
4.00
23.08
13.40
1.20
9.49
2.30
10.68
3.60
15.38
10.43
1.84
5.21
12.60
Claude-4-Sonnet
6.67
2.93
8.33
4.65
4.00
5.60
0.00
8.06
2.10
25.00
6.33
0.64
4.86
3.70
Claude-4.5-Sonnet
3.16
3.17
16.67
3.51
1.05
4.14
0.00
14.49
4.82
36.36
10.00
3.28
8.74
10.20
Claude-4.6-Sonnet
59.66
43.60
73.33
55.04
43.52
54.01
35.71
63.86
51.24
86.68
60.62
49.41
56.92
55.77
Figure 6: Example of error that occured with the model OpenAI CUA.
Table 5: Lifecycle grouping based on DS-World task category hierarchy.
Category
Lifecycle / Macro Group
Data Loading and Multi-Table Joining
Data Acquisition
Web Data Scraping
Data Acquisition
Summary Statistics
EDA
Data Cleaning and Missing-Value Handling
EDA
Data Filtering and Conditional Queries
EDA
Correlation and Relationship Analysis
EDA
Grouping and Aggregation
EDA
Outlier Detection
EDA
Feature Engineering and Transformation
Feature Engineering
Dimensionality Reduction
Feature Engineering
Standardization and Normalization
Feature Engineering
Feature Importance and Interpretability
Feature Engineering
Classification
Modeling
Regression
Modeling
Clustering
Modeling
Ensemble Methods
Modeling
Imbalanced Data Handling
Modeling
Model Validation and Evaluation
Evaluation and Deployment
Hyperparameter Tuning
Evaluation and Deployment
Statistical Testing
Evaluation and Deployment
Visualization and Chart Generation
Visualization and Reporting
Data Export and Reporting
Visualization and Reporting
Figure 7: Example of error that occured with the model Jedi-3B.
Table 6: Representative simplified Example Tasks in DSAgentBench by Category
Task Category
# Tasks
Example Task
Data Acquisition
23
Open the database, inspect the available tables and relevant columns, identify the customer ID and transaction fields, merge the required records, compute total and average transaction amounts per customer, calculate a risk score, and save the aggregated results to an output file.
Exploratory Data Analysis
119
Inspect a SQLite database schema, identify meal- and workout-related fields, join the relevant tables, compute an efficiency index, rank workout categories, and save the top five results to a CSV file.
Feature Engineering
37
Load a diabetes dataset, create a new insulin resistance feature, handle missing or zero values using median imputation, compute correlation with insulin levels, group results by outcome, and save summary statistics to output files.
Modeling
41
Load and merge multiple retail datasets by store and date, train multiple regression models to predict weekly sales, stack them using a meta-learner, perform cross-validation, and save the average error metric to a summary file.
Evaluation and Deployment
12
Perform multi-stage hyperparameter tuning for a gradient boosting model using cross-validation, select the best configuration based on error metrics, and save the final model parameters and performance results.
Visualization and Reporting
33
Analyze the relationship between body mass index, glucose level, and age with diabetes outcome, generate and inspect interactive visualizations, and create a PowerPoint report summarizing the key findings with appropriate charts, labels, and legends..
Figure 8: Example of error that occured with the model Jedi-8B.
Table 7: Canonical agent prompt templates used under different observation modalities. One representative prompt is shown per setting; other variants differ only in minor wording or formatting.
Observation Modality
Output Format
Prompt Description
Screenshot-only
Code / Action
The agent receives a full-resolution screenshot of the desktop at each step and predicts grounded mouse and keyboard actions based solely on visual information. Interaction is coordinate-based, without access to image matching or additional screenshots. At each step, the agent must return either executable actions or a control token (WAIT, FAIL, DONE).
Screenshot + A11y Tree
Code / Action
The agent jointly observes both the screenshot and the accessibility tree extracted via AT-SPI, enabling alignment between visual context and structured UI metadata. This hybrid prompt improves grounding precision and interaction robustness and is used for all main experiments unless stated otherwise.
Figure 9: Example of error that occured with the model UI-Tars-1.5-7B.
Table 8: Unified action space listing specific commands and their functions.
Action Category
Description
Mouse Actions
Handles spatial interactions including absolute cursor movement (MOVE_TO, DRAG_TO), scrolling (SCROLL), and button operations (CLICK, RIGHT_CLICK, DOUBLE_CLICK, MOUSE_DOWN, MOUSE_UP).
Keyboard Actions
Manages alphanumeric input via TYPING, discrete key presses (PRESS), simultaneous key combinations (HOTKEY), and raw key state manipulation (KEY_DOWN, KEY_UP).
Control Actions
Special meta-actions to pause execution (WAIT) or terminate the episode by signaling failure (FAIL) or success (DONE).
Table 9: Efficiency of successful runs measured by interaction steps.
Model
Mean Steps
Median Steps
Std
CUA
15.00
15
0.00
GPT-4.0
10.03
15
7.63
Gemini-2.5-Pro
6.76
6
2.93
GPT-5-Mini
7.33
5
3.82
Claude-Sonnet-4.6
10.93
12
3.04
Table 10: Root-cause failure analysis across models (%).
Model
Grounding
Terminal
Code
Logic
CUA
94.44
2.78
0.46
2.31
GPT-4.0
80.00
5.64
13.33
1.03
Gemini-2.5-Pro
43.08
36.15
13.85
6.92
GPT-5-Mini
56.92
7.69
30.77
4.62
GPT-5
41.67
8.33
38.33
11.67
Claude-Sonnet-4.5
39.32
23.07
27.60
10.00
Claude-Sonnet-4.6
32.77
9.24
43.70
14.29
GUI-OWL-7B
97.09
1.09
1.45
0.36
UI-Tars-1.5-7B
98.18
1.82
0.00
0.00
Jedi-7B
81.89
6.69
7.09
4.33
Table 11: Ablation results for GPT-4o on DSAgentBench under different interaction step budgets.
Metric
15 Steps
30 Steps
50 Steps
Task Success Rate (%)
24.54
25.45
25.81
Average Score
0.55
0.56
0.57
Table 12: Model-wise performance and trajectory behavior (normalized over 100 tasks per model for comparison). FF denotes the first failure step, i.e., the earliest step in the trajectory where an error is detected.
Model
#Tasks
Mean FF Step
Median FF
Early (%)
Mid (%)
Late (%)
Budget Exhaustion (%)
Recovery (%)
CUA
65
13.62
15
3.41
3.41
93.17
100.0
0.0
GPT-4.0
100
8.77
15
41.54
3.59
54.87
93.85
1.54
Gemini-2.5-Pro
100
5.80
6
42.02
45.38
12.61
62.31
19.23
GPT-5-Mini
100
5.92
4
57.38
19.67
22.95
63.08
3.08
GUI-OWL-7B
50
14.16
15
2.91
4.00
93.09
100.0
0.0
UI-Tars-1.5-7B
50
14.40
15
0.00
0.00
100.00
100.0
0.73
Jedi-7B
50
12.10
15
11.02
18.11
70.87
98.43
1.18
Table 13: DSAgentBench accuracy (%) across data-science lifecycle task types and overall performance under Screenshot settings (Open-Source Models).
Model
Screenshot
Screenshot + A11y Tree
DA
EDA
FE
Model
Vis
Eval
Overall
DA
EDA
FE
Model
Vis
Eval
Overall
Jedi-3B w/GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Jedi-7B w/GPT4o
0.84
0.00
0.00
0.00
1.00
0.00
0.73
N/A
N/A
N/A
N/A
N/A
N/A
N/A
UI-Tars-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
UI-Tars-1.5-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GUI-OWL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
OpenCUA-72B
4.35
0.91
0.00
0.00
0.00
0.00
0.73
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Table 14: Structure of a deterministic numerical evaluation function.
Evaluation Step
Description
Script Validation
Confirms that the required Python script exists and executes successfully.
Output Verification
Verifies that the expected output file (e.g., .txt or .csv) is generated.
Value Extraction
Extracts numeric values using robust parsing to avoid formatting artifacts.
Numerical Matching
Compares extracted values against ground-truth references using a small tolerance to account for floating-point variation.
Scoring
Assigns partial credit for intermediate correctness and caps the final score at 1.0.
Table 15: DSAgentBench average score (%) across ds lifecycle task types and overall performance under Screenshot and Screenshot + Accessibility Tree observation settings.
Figure 2: Benchmark Construction Pipeline: We source heterogeneous real-world datasets (1); design tasks and evaluators through human–LLM collaboration with pre- and post-execution configurations (2); and apply dual-annotator review and verification (3). The result is a curated benchmark of reproducible tasks with complete configurations and deterministic evaluation scripts.
Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.
