SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
Researchers built a new test called SWE-bench Science, made of 119 repair tasks pulled from 98 real GitHub repositories across 20 scientific fields like chemistry, biology, and physics. Even the top-performing AI coding agent, Claude Code paired with Opus-5, solved fewer than half the tasks correctly under strict private testing. The team also found that feeding agents extra scientific background information doesn't always help, and can sometimes hurt accuracy.
METAL MEDIA explanatory visual
AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- 01Unlike older coding benchmarks that test writing short functions or fixing generic software bugs, this one tests whether AI agents can repair real scientific research code, like simulations or instrument data pipelines, while preserving scientific correctness
- 02Tasks are split into three types: Issue-driven (fix a known bug), Expert-exploratory (figure out an unknown root cause like a scientist would), and Engineering-integration (connect multiple modules into a working pipeline)
- 03Eight AI agent setups were tested, including GPT-5.6-sol, Claude-Opus-5, and DeepSeek-V4-Pro; they scored well over 90% on visible public checks but their success rate dropped sharply on hidden, stricter private tests
- 04Failures were grouped into four recurring patterns: lacking scientific knowledge, shallow fixes that only patch symptoms, incomplete fixes that miss system-wide effects, and failing to generalize a scientific principle beyond the exact case shown
- 05In a follow-up test, giving agents extra scientific explanations helped one model (DeepSeek-V4-flash) but actually lowered the exact-success rate of another (GPT-5.6-sol), showing that more information doesn't automatically mean better repairs
What they did
- Unlike older coding benchmarks that test writing short functions or fixing generic software bugs, this one tests whether AI agents can repair real scientific research code, like simulations or instrument data pipelines, while preserving scientific correctness
- Tasks are split into three types: Issue-driven (fix a known bug), Expert-exploratory (figure out an unknown root cause like a scientist would), and Engineering-integration (connect multiple modules into a working pipeline)
- Eight AI agent setups were tested, including GPT-5.6-sol, Claude-Opus-5, and DeepSeek-V4-Pro; they scored well over 90% on visible public checks but their success rate dropped sharply on hidden, stricter private tests
- Failures were grouped into four recurring patterns: lacking scientific knowledge, shallow fixes that only patch symptoms, incomplete fixes that miss system-wide effects, and failing to generalize a scientific principle beyond the exact case shown
- In a follow-up test, giving agents extra scientific explanations helped one model (DeepSeek-V4-flash) but actually lowered the exact-success rate of another (GPT-5.6-sol), showing that more information doesn't automatically mean better repairs
| Scientific domain | Tasks |
|---|---|
| Chemistry | 24 |
| Materials Science and Engineering | 16 |
| Biology | 13 |
| Biomedical Engineering | 12 |
| Physics | 11 |
| Mathematics | 7 |
| Astronomy | 7 |
| Atmospheric Science | 5 |
| Civil Engineering | 5 |
| Surveying and Mapping Science and Technology | 3 |
| Geophysics | 3 |
| Mechanics | 3 |
| Electrical Engineering | 2 |
| Marine Science | 2 |
| Geography | 1 |
| Aeronautical and Astronautical Science and Technology | 1 |
| Nuclear Science and Technology | 1 |
| Computer Science and Technology | 1 |
| Statistics | 1 |
| Information and Communication Engineering | 1 |
| Total | 119 |
Why it matters
As AI increasingly writes and fixes the code behind scientific results, a broken patch can quietly corrupt the evidence behind a published finding, not just crash a program. This work gives concrete evidence that current AI agents often pass surface-level checks while missing deeper scientific correctness, which matters for anyone building or relying on AI coding tools in research.
Terms in this paper
- pass@1 · a strict success score counting a task solved only if every hidden private test passes on the first try
- Issue-driven task · a task built from an actual reported bug that the agent must fix
- Expert-exploratory task · a task where the agent must independently investigate an unknown cause, like a scientist debugging a puzzling result
- Engineering-integration task · a task requiring the agent to connect multiple files and modules into a complete working feature
- public/private tests · public tests are visible to the agent for debugging, private tests are hidden and only run after submission to judge true success
Figures we cannot republish
- Figure 1
Original abstract (English)
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Read on arXivLatest papers
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right call
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- SafeBranch: Branch-Pair Safety Alignment for Embodied AgentsTeaching a robot AI to know exactly which past decision was unsafe, by rewinding the sim to that moment and showing both choices side by side
Latest from METAL MEDIA
Figures: Zhipeng Xu et al., arXiv:2608.19799, CC BY 4.0