Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
arXiv:2608.103662026-08-10
AI agents were asked to run full data-science projects on a real computer, and even the best one only succeeded 57% of the time
DSAgentBench is a 275-task benchmark that tests whether AI agents can carry out an entire data-science workflow, from data acquisition to visualization and model evaluation, inside a real operating system using notebooks, IDEs, terminals, browsers, and databases. Testing 15 closed- and open-source models, the best performer, Claude-4.6-Sonnet, reached only 56.70% task success, while every open-source agent stayed below 1%. Human participants under the same setup reached 85.09%, exposing a large gap between current agents and real workflows.
METAL MEDIA explanatory visual
How DSAgentBench Evaluates Agents
Evidence statusMeasured results reported
1. Build a real desktop environmentExtended OSWorld on Ubuntu with VS Code, Jupyter Notebook, Chrome, Kaggle API, OpenML, and SQLite pre-installed
2. Design 275 tasksFour expert annotators spent three months creating and dual-verifying tasks from real Kaggle/OpenML/SQLite/GitHub data across six lifecycle stages
3. Agent perception-action loopThe agent views a screenshot (plus accessibility tree) and performs mouse/keyboard actions for up to 15 steps to complete the task
4. Deterministic scoringA script checks final outputs, not just code execution, such as numeric accuracy, chart labels/legends, and model performance
5. Compare resultsClaude-4.6-Sonnet scored 56.70% versus humans' 85.09%, while every open-source agent stayed below 1%, exposing the capability gap
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Prior benchmarks mainly check whether generated code runs, not whether an agent can actually operate a computer (navigating files, switching tools, recovering from errors), so the authors extended the OSWorld desktop-control framework specifically for data science.
Four expert annotators spent three months building and dual-verifying 275 tasks drawn from real datasets on Kaggle, OpenML, SQLite, and GitHub, spanning six stages: data acquisition, exploratory analysis, feature engineering, modeling, evaluation, and visualization.
Agents perceive the desktop through screenshots and an accessibility tree (structured information about on-screen buttons and elements) and act with mouse and keyboard; a deterministic script then checks the actual outputs, such as correlation values, chart labels/legends, and model accuracy, rather than just whether code executed.
Claude-4.6-Sonnet led with 56.70% success under the screenshot-plus-accessibility-tree setting, followed by GPT-5 at 29.81%; other closed models (GPT-4o, Gemini-2.5-Pro, GPT-5-mini) hovered around 20%, and all open-source agents stayed under 1%, while three human participants reached 85.09%.
Raising the maximum allowed steps from 15 to 50 barely improved success rate (24.54% to 25.81%), indicating the bottleneck is not the number of attempts but the agents' underlying perception, planning, and reasoning ability.
Figure 1: Example agent workflow task in DSAgentBench, showing selected stages from a multi-step workflow: the agent retrieves data, executes code, and produces the final outputs. The illustrated trajectory is generated by GPT-4o.
Table 1: Comparison of DSAgentBench with existing benchmarks across unified agent and data-science capabilities. ✓ = Yes, ▲ = Partial, ✗ = No.
Benchmark
Full OS Interaction
Web Tools & Browsers
Terminal + GUI Control
Cross-App Usage
Intermediate State
Controlled Exec Env
Multimodal Support
Data-Science Tasks
Visualization Evaluation
Environment Scalability
HumanEval chen2021evaluating
✗
✗
✗
✗
✗
✗
✗
✗
✗
✗
KRAMABench lai2025kramabench
✗
✗
✗
✗
✓
code
✗
▲
✗
✗
DS-1000 lai2023ds
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DABStep egg2025dabstep
✗
✗
✗
✗
✓
code
✗
▲
✗
✗
MLAgentBench huang2023mlagentbench
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DSBench jing2024dsbench
✗
✗
✗
✗
✗
✗
✓
▲
✗
✗
DSEval zhang2024benchmarking
✗
✗
✗
✗
✗
✓
✗
▲
✗
✗
ARCADE yin2023natural
✗
✗
✗
✗
✗
✗
✗
▲
✗
✗
DA-CODE huang2024code
✗
✗
✗
✗
✗
code
✗
▲
▲
▲
OSWORLD xie2024osworld
✓
✓
✓
✓
✓
✓
✓
✗
✗
✓
DSAgentBench (Ours)
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Figure 3: Task category distribution in our benchmark, reflecting real-world data science practice.
Figure 4: DSAgentBench execution workflow: the agent perceives screenshots and the A11y tree, executes GUI-based actions, and is evaluated by a custom script.
Table 3: DSAgentBench accuracy (%) under two observation settings (Screenshot and Screenshot + A11y Tree) across data-science lifecycle tasks: Data Acquisition (DA), Exploratory Data Analysis (EDA), Feature Engineering (FE), Modeling, Visualization (Vis), and Evaluation (Eval). Results for open-source models are reported in Table 13.
Model
Screenshot
Screenshot + A11y Tree
DA
EDA
FE
Model
Vis
Eval
Overall
DA
EDA
FE
Model
Vis
Eval
Overall
GPT-4o
0.00
27.06
24.24
16.67
7.14
33.33
19.34
13.04
29.66
35.14
21.95
15.62
8.33
24.54
O4-mini
4.35
1.68
2.70
0.00
3.03
0.00
1.82
0.00
4.20
5.41
0.00
0.00
0.00
2.55
GPT-5-mini
8.70
17.28
25.00
3.57
21.43
0.00
15.20
4.55
24.14
27.03
7.50
21.88
0.00
19.03
GPT-5
17.39
27.41
27.03
26.83
9.09
16.67
23.63
26.08
33.44
29.72
32.22
15.15
33.33
29.81
Gemini-2.5-Pro
4.55
17.86
16.13
17.24
10.71
16.67
14.49
4.35
22.78
33.33
14.81
30.77
0.00
20.81
OpenAI CUA
0.00
10.59
12.50
0.00
14.29
0.00
8.13
4.35
9.52
6.06
2.86
6.67
0.00
6.61
Claude-4-Sonnet
0.00
7.46
4.17
0.00
3.85
0.00
4.55
0.00
4.86
0.00
1.00
11.11
0.00
4.64
Claude-4.5-Sonnet
4.76
4.17
0.00
3.45
3.23
0.00
3.17
8.33
12.31
12.50
4.55
4.17
0.00
9.21
Claude-4.6-Sonnet
43.48
57.98
51.35
41.46
39.39
50.00
50.55
47.82
64.88
56.75
46.34
42.42
66.67
56.70
Human Performance
73.91
90.76
91.89
73.17
78.79
83.33
85.09
73.91
90.76
91.89
73.17
78.79
83.33
85.09
Figure 5: Example environments in DSAgentBench: agent interaction across data-science environments, demonstrating autonomous coordination of file systems, databases, notebooks, IDEs, visualization, execution, debugging, and web-based data retrieval.
Table 4: DSAgentBench accuracy (%) across stage type, task complexity, and tool usage under different settings.
Model
Screenshot
Screenshot + A11y Tree
Single
Multi
Easy
Medium
Hard
VS Code
Jupyter
Single
Multi
Easy
Medium
Hard
VS Code
Jupyter
GPT-4o
26.97
13.52
38.46
24.74
11.79
14.69
39.22
35.59
16.13
46.67
34.38
12.31
23.87
27.45
O4-mini
4.20
0.00
20.00
1.55
0.00
1.79
1.92
5.04
0.64
13.33
3.88
0.00
2.69
1.92
GPT-5-mini
24.71
8.00
38.46
24.47
9.41
15.23
11.00
31.03
9.87
66.67
27.20
5.47
14.68
38.00
GPT-5
27.73
20.51
53.33
27.13
16.79
25.10
17.30
39.49
22.43
66.67
32.55
22.89
27.35
40.38
Gemini-2.5-Pro
19.12
10.92
46.15
15.62
9.18
17.31
2.40
32.10
12.93
53.85
28.74
9.28
17.45
31.25
OpenAI CUA
13.64
4.00
23.08
13.40
1.20
9.49
2.30
10.68
3.60
15.38
10.43
1.84
5.21
12.60
Claude-4-Sonnet
6.67
2.93
8.33
4.65
4.00
5.60
0.00
8.06
2.10
25.00
6.33
0.64
4.86
3.70
Claude-4.5-Sonnet
3.16
3.17
16.67
3.51
1.05
4.14
0.00
14.49
4.82
36.36
10.00
3.28
8.74
10.20
Claude-4.6-Sonnet
59.66
43.60
73.33
55.04
43.52
54.01
35.71
63.86
51.24
86.68
60.62
49.41
56.92
55.77
Figure 6: Example of error that occured with the model OpenAI CUA.
Table 5: Lifecycle grouping based on DS-World task category hierarchy.
Category
Lifecycle / Macro Group
Data Loading and Multi-Table Joining
Data Acquisition
Web Data Scraping
Data Acquisition
Summary Statistics
EDA
Data Cleaning and Missing-Value Handling
EDA
Data Filtering and Conditional Queries
EDA
Correlation and Relationship Analysis
EDA
Grouping and Aggregation
EDA
Outlier Detection
EDA
Feature Engineering and Transformation
Feature Engineering
Dimensionality Reduction
Feature Engineering
Standardization and Normalization
Feature Engineering
Feature Importance and Interpretability
Feature Engineering
Classification
Modeling
Regression
Modeling
Clustering
Modeling
Ensemble Methods
Modeling
Imbalanced Data Handling
Modeling
Model Validation and Evaluation
Evaluation and Deployment
Hyperparameter Tuning
Evaluation and Deployment
Statistical Testing
Evaluation and Deployment
Visualization and Chart Generation
Visualization and Reporting
Data Export and Reporting
Visualization and Reporting
Figure 7: Example of error that occured with the model Jedi-3B.
Table 6: Representative simplified Example Tasks in DSAgentBench by Category
Task Category
# Tasks
Example Task
Data Acquisition
23
Open the database, inspect the available tables and relevant columns, identify the customer ID and transaction fields, merge the required records, compute total and average transaction amounts per customer, calculate a risk score, and save the aggregated results to an output file.
Exploratory Data Analysis
119
Inspect a SQLite database schema, identify meal- and workout-related fields, join the relevant tables, compute an efficiency index, rank workout categories, and save the top five results to a CSV file.
Feature Engineering
37
Load a diabetes dataset, create a new insulin resistance feature, handle missing or zero values using median imputation, compute correlation with insulin levels, group results by outcome, and save summary statistics to output files.
Modeling
41
Load and merge multiple retail datasets by store and date, train multiple regression models to predict weekly sales, stack them using a meta-learner, perform cross-validation, and save the average error metric to a summary file.
Evaluation and Deployment
12
Perform multi-stage hyperparameter tuning for a gradient boosting model using cross-validation, select the best configuration based on error metrics, and save the final model parameters and performance results.
Visualization and Reporting
33
Analyze the relationship between body mass index, glucose level, and age with diabetes outcome, generate and inspect interactive visualizations, and create a PowerPoint report summarizing the key findings with appropriate charts, labels, and legends..
Figure 8: Example of error that occured with the model Jedi-8B.
Table 7: Canonical agent prompt templates used under different observation modalities. One representative prompt is shown per setting; other variants differ only in minor wording or formatting.
Observation Modality
Output Format
Prompt Description
Screenshot-only
Code / Action
The agent receives a full-resolution screenshot of the desktop at each step and predicts grounded mouse and keyboard actions based solely on visual information. Interaction is coordinate-based, without access to image matching or additional screenshots. At each step, the agent must return either executable actions or a control token (WAIT, FAIL, DONE).
Screenshot + A11y Tree
Code / Action
The agent jointly observes both the screenshot and the accessibility tree extracted via AT-SPI, enabling alignment between visual context and structured UI metadata. This hybrid prompt improves grounding precision and interaction robustness and is used for all main experiments unless stated otherwise.
Figure 9: Example of error that occured with the model UI-Tars-1.5-7B.
Table 8: Unified action space listing specific commands and their functions.
Action Category
Description
Mouse Actions
Handles spatial interactions including absolute cursor movement (MOVE_TO, DRAG_TO), scrolling (SCROLL), and button operations (CLICK, RIGHT_CLICK, DOUBLE_CLICK, MOUSE_DOWN, MOUSE_UP).
Keyboard Actions
Manages alphanumeric input via TYPING, discrete key presses (PRESS), simultaneous key combinations (HOTKEY), and raw key state manipulation (KEY_DOWN, KEY_UP).
Control Actions
Special meta-actions to pause execution (WAIT) or terminate the episode by signaling failure (FAIL) or success (DONE).
Table 9: Efficiency of successful runs measured by interaction steps.
Model
Mean Steps
Median Steps
Std
CUA
15.00
15
0.00
GPT-4.0
10.03
15
7.63
Gemini-2.5-Pro
6.76
6
2.93
GPT-5-Mini
7.33
5
3.82
Claude-Sonnet-4.6
10.93
12
3.04
Table 10: Root-cause failure analysis across models (%).
Model
Grounding
Terminal
Code
Logic
CUA
94.44
2.78
0.46
2.31
GPT-4.0
80.00
5.64
13.33
1.03
Gemini-2.5-Pro
43.08
36.15
13.85
6.92
GPT-5-Mini
56.92
7.69
30.77
4.62
GPT-5
41.67
8.33
38.33
11.67
Claude-Sonnet-4.5
39.32
23.07
27.60
10.00
Claude-Sonnet-4.6
32.77
9.24
43.70
14.29
GUI-OWL-7B
97.09
1.09
1.45
0.36
UI-Tars-1.5-7B
98.18
1.82
0.00
0.00
Jedi-7B
81.89
6.69
7.09
4.33
Table 11: Ablation results for GPT-4o on DSAgentBench under different interaction step budgets.
Metric
15 Steps
30 Steps
50 Steps
Task Success Rate (%)
24.54
25.45
25.81
Average Score
0.55
0.56
0.57
Table 12: Model-wise performance and trajectory behavior (normalized over 100 tasks per model for comparison). FF denotes the first failure step, i.e., the earliest step in the trajectory where an error is detected.
Model
#Tasks
Mean FF Step
Median FF
Early (%)
Mid (%)
Late (%)
Budget Exhaustion (%)
Recovery (%)
CUA
65
13.62
15
3.41
3.41
93.17
100.0
0.0
GPT-4.0
100
8.77
15
41.54
3.59
54.87
93.85
1.54
Gemini-2.5-Pro
100
5.80
6
42.02
45.38
12.61
62.31
19.23
GPT-5-Mini
100
5.92
4
57.38
19.67
22.95
63.08
3.08
GUI-OWL-7B
50
14.16
15
2.91
4.00
93.09
100.0
0.0
UI-Tars-1.5-7B
50
14.40
15
0.00
0.00
100.00
100.0
0.73
Jedi-7B
50
12.10
15
11.02
18.11
70.87
98.43
1.18
Table 13: DSAgentBench accuracy (%) across data-science lifecycle task types and overall performance under Screenshot settings (Open-Source Models).
Model
Screenshot
Screenshot + A11y Tree
DA
EDA
FE
Model
Vis
Eval
Overall
DA
EDA
FE
Model
Vis
Eval
Overall
Jedi-3B w/GPT4o
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Jedi-7B w/GPT4o
0.84
0.00
0.00
0.00
1.00
0.00
0.73
N/A
N/A
N/A
N/A
N/A
N/A
N/A
UI-Tars-2B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
UI-Tars-1.5-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GUI-OWL-7B
0.00
0.00
0.00
0.00
0.00
0.00
0.00
N/A
N/A
N/A
N/A
N/A
N/A
N/A
OpenCUA-72B
4.35
0.91
0.00
0.00
0.00
0.00
0.73
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Table 14: Structure of a deterministic numerical evaluation function.
Evaluation Step
Description
Script Validation
Confirms that the required Python script exists and executes successfully.
Output Verification
Verifies that the expected output file (e.g., .txt or .csv) is generated.
Value Extraction
Extracts numeric values using robust parsing to avoid formatting artifacts.
Numerical Matching
Compares extracted values against ground-truth references using a small tolerance to account for floating-point variation.
Scoring
Assigns partial credit for intermediate correctness and caps the final score at 1.0.
Table 15: DSAgentBench average score (%) across ds lifecycle task types and overall performance under Screenshot and Screenshot + Accessibility Tree observation settings.
Ensures that the visualization script exists and executes successfully.
Artifact Verification
Confirms that required visualization files are generated and non-empty.
Semantic Validation
Uses a fixed evaluation prompt with GPT-4o as a judge to assess whether each chart correctly represents the intended variables, axes, and trends.
Design Criteria
Checks readability, labeling, scale consistency, and layout clarity.
Scoring
Combines deterministic and semantic signals into a bounded score in the range [0, 1].
Findings
The top model, Claude-4.6-Sonnet, reached 56.70% task success under screenshot-plus-accessibility-tree observation, followed by GPT-5 at 29.81%, while GPT-4o, Gemini-2.5-Pro, and GPT-5-mini scored around 20%.
All open-source agents tested (UI-TARS, Jedi, GUI-OWL, OpenCUA, etc.) scored below 1% success, and none of them supported accessibility-tree input.
Three human participants (two applied scientists and one master's graduate) reached an 85.09% success rate under the identical environment and evaluation protocol, far exceeding any tested agent.
For open-source agents, 97-98% of failures were grounding errors (failing to correctly perceive the UI state), while stronger closed models like Gemini-2.5-Pro, GPT-5, and Claude-4.6-Sonnet showed a mix of terminal, code, and reasoning failures.
Increasing the step budget from 15 to 50 only raised GPT-4o's success rate from 24.54% to 25.81%, and prompting it to prefer terminal-first workflows only moved the rate from 19.34% to 20.73%, showing that neither more attempts nor a different interaction style substantially closes the gap.
Where it can be used
Can serve as a standard yardstick for how ready an AI agent is to automate real-world data-analysis work.
Can be used to diagnose separately whether an agent's weakness lies in perceiving the screen, coordinating multiple applications, or recovering from errors.
Can act as a comparison baseline (against the 85.09% human success rate) when developing new GUI-control or data-science-specialized agents.
Limits and open work
All experiments were run on Ubuntu only; performance on Windows or macOS environments has not been measured, even though the framework claims portability.
Tasks are capped at 15 interaction steps and a 1800-second timeout, so the benchmark does not cover much longer real-world workflows.
Only about 10% of tasks use an LLM-based visual judge for evaluation, with the rest relying on rule-based checks, limiting how broadly the visualization-quality assessment generalizes.
Extensibility to tools like R/RStudio, PyCharm, or cloud environments is mentioned as a design possibility but was not actually tested in this paper.
The authors state plans to release prompts, evaluation code, and inference scripts, but this is a future commitment, not part of the reported results.
Why it matters
Real workplace data analysis rarely means writing one perfect script; it means bouncing between programs, fixing errors, and checking intermediate outputs, and this benchmark shows concrete numbers on how far current AI agents are from that reality. It gives developers a measurable baseline (human success at 85.09%) to know which specific capabilities—screen grounding, tool switching, multi-step reasoning—need improvement before deploying agents on real analytical work.
Terms in this paper
OSWorld · An existing benchmark framework that tests whether AI can control real operating systems (Ubuntu, Windows, macOS) with mouse and keyboard
Accessibility (A11y) Tree · Structured data listing what buttons, text, and windows are on screen, originally built for assistive technology, now used to help AI locate UI elements
Deterministic evaluator · An automatic scoring script that gives the same score for the same output every time, without relying on human or subjective judgment
Task success rate · The share of tasks scoring 0.95 or higher, meaning the output must be almost perfectly correct to count as a success
Figures we cannot republish
Figure 2: Benchmark Construction Pipeline: We source heterogeneous real-world datasets (1); design tasks and evaluators through human–LLM collaboration with pre- and post-execution configurations (2); and apply dual-annotator review and verification (3). The result is a curated benchmark of reproducible tasks with complete configurations and deterministic evaluation scripts.
Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.
