A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
只改变量名不改逻辑,AI编程智能体照样会翻车
研究者测试了修复真实软件缺陷的AI编程智能体,在代码被改写成外观不同但行为完全一致的版本后是否仍然可靠。他们对代码库做了一批只改外观、不改实际运行结果的重写,然后让智能体在原始代码和改写代码上各跑多次,比较修复成功率。结果显示大多数组合都出现了小幅性能下降,而哪个模型'最稳健'完全取决于用的智能体框架和测试基准。
METAL MEDIA 解读图
只改变量名不改逻辑,AI编程智能体照样会翻车
- 01研究团队设计了14种'语义保持变换'——比如改写控制流、插入不影响结果的死代码、给变量和函数改名——这些改动只改变代码外观,不改变代码实际行为(在数万条测试用例上验证过)。
- 02他们在Claude Opus 4.5、Kimi K2.5、MiniMax M2.5、Qwen 3.6-27B四个前沿模型上分别搭配mini-SWE agent和OpenCode两种智能体框架,在SWE-bench Verified和SWE-bench Pro两个基准共54个真实修复任务上做了测试。
- 03每个任务上,智能体分别在原始代码和改写后的变体代码上各跑20次,通过配对比较把改写带来的影响和智能体自身运行的随机性区分开来。
- 0416种模型-框架-基准组合里有6种出现了具有统计显著性的成功率下降(最高达6.7个百分点),即便成功率没怎么变,智能体所用的步骤数和花费(token成本)也可能分别多出9.9%和22.9%。
- 05没有哪个模型能一直保持最稳健:Qwen在mini-SWE agent下表现最稳,换到OpenCode下却变得最脆弱;总体上更简单的mini-SWE agent框架比OpenCode更稳健。
他们做了什么
- 研究团队设计了14种'语义保持变换'——比如改写控制流、插入不影响结果的死代码、给变量和函数改名——这些改动只改变代码外观,不改变代码实际行为(在数万条测试用例上验证过)。
- 他们在Claude Opus 4.5、Kimi K2.5、MiniMax M2.5、Qwen 3.6-27B四个前沿模型上分别搭配mini-SWE agent和OpenCode两种智能体框架,在SWE-bench Verified和SWE-bench Pro两个基准共54个真实修复任务上做了测试。
- 每个任务上,智能体分别在原始代码和改写后的变体代码上各跑20次,通过配对比较把改写带来的影响和智能体自身运行的随机性区分开来。
- 16种模型-框架-基准组合里有6种出现了具有统计显著性的成功率下降(最高达6.7个百分点),即便成功率没怎么变,智能体所用的步骤数和花费(token成本)也可能分别多出9.9%和22.9%。
- 没有哪个模型能一直保持最稳健:Qwen在mini-SWE agent下表现最稳,换到OpenCode下却变得最脆弱;总体上更简单的mini-SWE agent框架比OpenCode更稳健。
| Transformation | Summary |
|---|---|
| If Else Switcher | Swaps if/else branches and negates the condition |
| For Loop Rewriting | Rewrites a for loop using an explicit iterator |
| And Condition Splitter | Decomposes if A and B into nested ifs |
| Comparison Swapper | Swaps operands and inverts the operator |
| While Loop Unrolling | Unrolls one iteration of a while loop |
| Double Negation Injector | Wraps a condition in not not (⋅) |
| Commutative Operand Permuter | Reorders commutative operands |
| Local Variable Renamer | Renames safe local variables to synonyms |
| If True Wrapper | Wraps a block in a permanently true guard |
| Try Except Injector | Wraps a block in a redundant try/except |
| Dead Code Injector | Inserts an unreachable block |
| Dead String Assignment | Inserts an unread variable assignment |
| Dead Method Injection | Appends an unreachable method to a class |
| String Literal Splitter | Splits a string literal into a concatenation |
| Category | mini-SWE | OpenCode |
|---|---|---|
| Architecture | Single, centralized agent | Primary agents invoke tools and subagents for specific tasks (e.g., Build, Plan, General, Explore) |
| Available Tools | bash | bash, edit, write, read, grep, glob, lsp, apply_patch, skill, todo_write, web_fetch, web_search, question |
| Access Restrictions | None | Available tools determined by an agent’s role and user configuration |
| Context Management | Continuous, append-only | Multiple, dynamically changing contexts for different subagents |
为什么重要
现实世界的代码库很少和测试基准里的代码长得一模一样,这项研究说明即便只是表面上、不改变实际功能的差异,也可能动摇顶尖AI编程智能体在部署环境中的表现。这提醒从业者,依据某个基准的稳健性排名选模型,换个框架或代码库后结果可能完全不同。
本文术语
- 语义保持变换(SPT) · 只改变代码外观、不改变代码实际运行结果的编辑方式,通过测试结果完全一致来验证
- 修复成功率(resolve rate) · 智能体多次尝试中真正成功修复缺陷的比例
- SWE-bench Verified / SWE-bench Pro · 用真实开源项目缺陷修复任务评估AI编程智能体能力的标准测试基准
- 智能体框架(agentic scaffold) · 让AI模型能够浏览、修改、测试代码库的程序骨架,如mini-SWE agent、OpenCode
- 死代码注入 · 插入运行了但不影响程序输出结果的代码,用来测试智能体是否会被它干扰
无法转载的图表
- Figure 1: Mean degradation Δ¯ (percentage points) for each model under each scaffold on both benchmarks. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 2: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right), restricted to runs the agent resolved in both cases. Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 3: Mean resolve rate (%) on unperturbed and perturbed runs for each model under each scaffold, on SWE-bench Verified (top) and SWE-bench Pro (bottom).
- Figure 4: Mean relative change in agent steps (δ¯step) and cost (δ¯cost) after perturbation over all runs, for each scaffold–model configuration on the 28 SWE-bench Verified instances (left) and the 26 SWE-bench Pro instances (right). Positive values mean perturbed runs take more steps, or cost more, than unperturbed runs on the same instance. Error bars are fixed-population 95% bootstrap confidence intervals.
- Figure 5: Per-instance degradation Δ(i) for the 28 SWE-bench Verified instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
- Figure 6: Per-instance degradation Δ(i) for the 26 SWE-bench Pro instances under each configuration. Error bars are Newcombe 95% confidence intervals for the difference between the unperturbed and perturbed resolve proportions (n=20 runs per condition).
论文原文摘要（英文）
AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调