SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution
让AI修bug代理在遇到真实问题前先给自己出'模拟考题'练手,从而提前吃透某个项目的特殊习惯
SkillForge让基于大模型的软件修复代理主动学习特定代码仓库的项目特性:它利用仓库自身的测试代码,让AI在看不到原始实现的情况下重写关键功能,从而制造出逼真的模拟bug并自己解决,再从解决过程中提炼出可复用的'技能'。以往的方法要么依赖大量历史issue记录,要么每次遇到新问题都要花费高昂代价临场探索。在SWE-bench Verified和SWE-bench Pro两个基准测试上,SkillForge的表现持续超过基于历史记录和基于在线探索的现有方法。
METAL MEDIA 解读图
让AI修bug代理在遇到真实问题前先给自己出'模拟考题'练手,从而提前吃透某个项目的特殊习惯
- 01找到仓库中真正被测试执行到的核心代码区域,让大模型在不看原始实现的情况下重新实现这部分功能,从而自然产生逼真的实现错误,并把由此导致的测试失败包装成模拟issue
- 02让一个软件修复代理去解决这些模拟issue,再从成功和失败的尝试中分别提炼出两类知识:关联到具体代码实体的全局诊断技能(该从哪里入手排查、涉及哪些相关API)和局部修改技能(该怎么改、要避开哪些项目特有的坑)
- 03真实issue到来时,先根据issue描述用BM25检索相关的全局诊断技能预先注入上下文,再在代理实际访问到对应文件或函数的那一刻,实时插入对应的局部修改建议
- 04在Mini-SWE-Agent上实现并用DeepSeek-V3.2和GPT-5-mini测试,SkillForge在SWE-bench Verified上的Pass@1相比基线分别提升5.8和5.6个百分点,在SWE-bench Pro上分别提升5.8和4.1个百分点,超过了所有对比的历史驱动和在线探索类方法
他们做了什么
- 找到仓库中真正被测试执行到的核心代码区域,让大模型在不看原始实现的情况下重新实现这部分功能,从而自然产生逼真的实现错误,并把由此导致的测试失败包装成模拟issue
- 让一个软件修复代理去解决这些模拟issue,再从成功和失败的尝试中分别提炼出两类知识:关联到具体代码实体的全局诊断技能(该从哪里入手排查、涉及哪些相关API)和局部修改技能(该怎么改、要避开哪些项目特有的坑)
- 真实issue到来时,先根据issue描述用BM25检索相关的全局诊断技能预先注入上下文,再在代理实际访问到对应文件或函数的那一刻,实时插入对应的局部修改建议
- 在Mini-SWE-Agent上实现并用DeepSeek-V3.2和GPT-5-mini测试,SkillForge在SWE-bench Verified上的Pass@1相比基线分别提升5.8和5.6个百分点,在SWE-bench Pro上分别提升5.8和4.1个百分点,超过了所有对比的历史驱动和在线探索类方法
| Method | Model | Pass@1 | Avg Cost |
|---|---|---|---|
| Mini-SWE-Agent | DeepSeek-V3.2 | 66.4% | $0.049 |
| GPT-5-mini | 55.0% | $0.031 | |
| History-driven project-specific knowledge acquisition | |||
| SWE-Exp | DeepSeek-V3.2 | 69.0%†↑2.6% | $0.090 |
| GPT-5-mini | 56.6%†↑1.6% | $0.065 | |
| EvoCoder | DeepSeek-V3.2 | 67.0% ↑0.6% | $0.064 |
| GPT-5-mini | 58.4% ↑3.4% | $0.052 | |
| MemGovern | DeepSeek-V3.2 | 69.2%†↑2.8% | – |
| GPT-5-mini | 58.0%†↑3.0% | – | |
| Online project-specific knowledge acquisition | |||
| SAGE | DeepSeek-V3.2 | 67.2% ↑0.8% | $0.081 |
| GPT-5-mini | 56.0%†↑1.0% | $0.052 | |
| SWE-Debate | DeepSeek-V3.2 | 68.2% ↑1.8% | $0.382 |
| GPT-5-mini | 56.4% ↑1.4% | $0.167 | |
| Live-SWE-agent | DeepSeek-V3.2 | 67.0% ↑0.6% | $0.050 |
| GPT-5-mini | 55.6% ↑0.6% | $0.042 | |
| Variants of SkillForge | |||
| SkillForge w/ SWE-Smith | DeepSeek-V3.2 | 68.0% ↑1.6% | $0.088 |
| GPT-5-mini | 56.4%†↑1.4% | $0.071 | |
| SkillForge w/ LLM Summary | DeepSeek-V3.2 | 68.7%†↑2.3% | $0.069 |
| GPT-5-mini | 54.4% ↓0.6% | $0.065 | |
| SkillForge | DeepSeek-V3.2 | 72.2%†↑5.8% | $0.074 |
| GPT-5-mini | 60.6%†↑5.6% | $0.066 | |
| †: p−value<0.05. |
| DeepSeek-V3.2 | GPT-5-mini | |||
|---|---|---|---|---|
| Method | Pass@1 | Avg Cost | Pass@1 | Avg Cost |
| Mini-SWE-Agent | 28.3% | $0.047 | 47.6% | $0.063 |
| SWE-Exp | 29.4% ↑1.1% | $0.083 | 48.7% ↑0.9% | $0.089 |
| Live-SWE-agent | 32.4% ↑4.1% | $0.051 | 49.1%†↑1.5% | $0.072 |
| SkillForge | 34.1%†↑5.8% | $0.069 | 51.7%†↑4.1% | $0.087 |
| †: p−value<0.05. |
| Approach | DeepSeek-V3.2 | GPT-5-mini |
|---|---|---|
| w/o Global Diagnostic Skills | 68.4% (↓3.8%) | 57.6% (↓3.0%) |
| w/o Local Intervention Skills | 67.8% (↓4.4%) | 57.2% (↓3.4%) |
| SkillForge | 72.2% | 60.6% |
| Resolver LLM | Knowledge-source LLM | Pass@1 |
|---|---|---|
| GPT-5-mini | GPT-5-mini | 60.6% |
| DeepSeek-V3.2 | 55.0% | |
| DeepSeek-V3.2 | GPT-5-mini | 65.2% |
| DeepSeek-V3.2 | 72.2% |
为什么重要
将AI修复代理部署到某个具体项目时,最大的障碍是它不了解该项目自身的编码习惯和常见陷阱,这种方法提供了一条即使历史issue数据稀缺也能提前为代理'补课'的实用途径,且不会大幅增加每个issue的处理成本。这对希望在自有代码仓库上落地自动修复代理的团队和开发者有直接参考价值。
本文术语
- Pass@1 · 一次尝试就成功解决问题的比例,用于衡量修复成功率
- SWE-bench · 基于真实GitHub issue评测AI编程代理修复能力的基准测试集
- BM25 · 一种根据关键词匹配程度对文档进行排序的经典信息检索算法
- 实体锚定(entity-grounded) · 把提炼出的知识直接关联到代码中具体的文件、函数或类,而不是笼统存放
- 自蒸馏(self-distillation) · 从代理自己生成的经验数据中提炼可复用知识的过程
论文原文摘要（英文）
Large language model (LLM) based agents have demonstrated remarkable proficiency in automated software issue resolution, yet they often struggle to resolve issues in a specific repository because they lack project-specific knowledge. Existing self-evolving approaches acquire such knowledge from repository history or online repair trajectories, but they either depend on available historical issue-resolution signals or incur substantial per-issue test-time exploration cost. In this paper, we propose SkillForge, a self-distillation framework that proactively acquires project-specific knowledge from the repository itself. Instead of waiting for real issues to expose project-specific knowledge gaps, SkillForge synthesizes project-specific issues by re-implementing test-covered core functionalities of the repository. By resolving these synthetic issues, SkillForge distills reusable project-specific knowledge into entity-grounded skills and associates them with relevant repository entities for future issue resolution. Extensive experiments using both open-source and closed-source models show that SkillForge consistently improves issue resolution performance over strong baselines. These results demonstrate that proactively acquiring project-specific knowledge before solving real issues substantially improves downstream software issue resolution.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调
METAL MEDIA 最新报道
图片来源： Silin Chen et al., arXiv:2608.18933, CC BY 4.0