A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
Rename a variable, keep the logic the same — AI coding agents still stumble
Researchers tested whether AI code agents that fix real software bugs stay reliable when the surrounding code is rewritten to look different but behave identically. They rewrote codebases with edits that preserve exact program behavior, then ran each agent repeatedly on both the original and rewritten versions to compare success rates. Most configurations showed small performance drops, and which model was 'most robust' completely flipped depending on the agent framework and benchmark used.
METAL MEDIA explanatory visual
Rename a variable, keep the logic the same — AI coding agents still stumble
- 01The team built 14 'semantics-preserving transformations' — edits like rewriting control flow, injecting inert dead code, or renaming identifiers — that change how code looks without changing what it actually does (verified against tens of thousands of tests per project).
- 02They tested four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B), each paired with two agent scaffolds (mini-SWE agent and OpenCode), on 54 real bug-fixing tasks drawn from SWE-bench Verified and SWE-bench Pro.
- 03For each task, the agent ran 20 times on the original code and 20 times on transformed variants, letting the researchers isolate the effect of the code rewrite from the agent's own run-to-run randomness.
- 046 of 16 model-scaffold-benchmark combinations showed statistically significant performance drops (up to 6.7 percentage points), and even when success rates held steady, agents often needed up to 9.9% more steps and 22.9% more cost (tokens) to finish.
- 05No model was consistently the most robust: Qwen was the sturdiest under mini-SWE agent but the most fragile under OpenCode, and the simpler mini-SWE agent scaffold was generally more robust than OpenCode overall.
What they did
- The team built 14 'semantics-preserving transformations' — edits like rewriting control flow, injecting inert dead code, or renaming identifiers — that change how code looks without changing what it actually does (verified against tens of thousands of tests per project).
- They tested four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B), each paired with two agent scaffolds (mini-SWE agent and OpenCode), on 54 real bug-fixing tasks drawn from SWE-bench Verified and SWE-bench Pro.
- For each task, the agent ran 20 times on the original code and 20 times on transformed variants, letting the researchers isolate the effect of the code rewrite from the agent's own run-to-run randomness.
- 6 of 16 model-scaffold-benchmark combinations showed statistically significant performance drops (up to 6.7 percentage points), and even when success rates held steady, agents often needed up to 9.9% more steps and 22.9% more cost (tokens) to finish.
- No model was consistently the most robust: Qwen was the sturdiest under mini-SWE agent but the most fragile under OpenCode, and the simpler mini-SWE agent scaffold was generally more robust than OpenCode overall.
| Transformation | Summary |
|---|---|
| If Else Switcher | Swaps if/else branches and negates the condition |
| For Loop Rewriting | Rewrites a for loop using an explicit iterator |
| And Condition Splitter | Decomposes if A and B into nested ifs |
| Comparison Swapper | Swaps operands and inverts the operator |
| While Loop Unrolling | Unrolls one iteration of a while loop |
| Double Negation Injector | Wraps a condition in not not (⋅) |
| Commutative Operand Permuter | Reorders commutative operands |
| Local Variable Renamer | Renames safe local variables to synonyms |
| If True Wrapper | Wraps a block in a permanently true guard |
| Try Except Injector | Wraps a block in a redundant try/except |
| Dead Code Injector | Inserts an unreachable block |
| Dead String Assignment | Inserts an unread variable assignment |
| Dead Method Injection | Appends an unreachable method to a class |
| String Literal Splitter | Splits a string literal into a concatenation |
| Category | mini-SWE | OpenCode |
|---|---|---|
| Architecture | Single, centralized agent | Primary agents invoke tools and subagents for specific tasks (e.g., Build, Plan, General, Explore) |
| Available Tools | bash | bash, edit, write, read, grep, glob, lsp, apply_patch, skill, todo_write, web_fetch, web_search, question |
| Access Restrictions | None | Available tools determined by an agent’s role and user configuration |
| Context Management | Continuous, append-only | Multiple, dynamically changing contexts for different subagents |
Why it matters
Real-world codebases rarely look exactly like benchmark code, so this shows that even superficial, meaning-preserving differences can shake top AI coding agents' performance in deployment. It warns practitioners that picking a model based on one benchmark's robustness ranking may not hold once the scaffold or codebase changes.
Terms in this paper
- Semantics-preserving transformation (SPT) · A code edit that changes how the code looks but not what it actually does, verified by identical test outcomes
- Resolve rate · The fraction of agent runs that successfully fix a given bug
- SWE-bench Verified / SWE-bench Pro · Standard benchmarks of real open-source bug-fixing tasks used to evaluate AI coding agents
- Agentic scaffold · The surrounding program (like mini-SWE agent or OpenCode) that lets an AI model explore, edit, and test code repositories
- Dead code injection · Adding code that runs but has no effect on the program's output, used to test if agents get distracted by it
Figures we cannot republish
- Figure 1: Mean degradation Δ¯ (percentage points) for each model under each scaffold on both benchmarks. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 2: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right), restricted to runs the agent resolved in both cases. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 3: Mean resolve rate (%) on unperturbed and perturbed runs for each model under each scaffold, on SWE-bench Verified (top) and SWE-bench Pro (bottom).
- Figure 4: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right). Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 5: Per-instance degradation Δ(i) for the 28 SWE-bench Verified instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
- Figure 6: Per-instance degradation Δ(i) for the 26 SWE-bench Pro instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
Original abstract (English)
AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears