SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution
An AI coding agent invents its own practice bugs before facing real ones, so it learns a project's quirks in advance
SkillForge lets an LLM-based software-repair agent proactively learn a specific repository's conventions by rewriting test-covered code to create realistic synthetic bugs, then solving them and distilling the experience into reusable 'skills.' Earlier approaches either needed a rich history of previously solved real issues or had to spend heavy computation exploring each new issue at the time it arrives. On SWE-bench Verified and SWE-bench Pro, SkillForge consistently beat prior history-based and online exploration baselines.
METAL MEDIA explanatory visual
An AI coding agent invents its own practice bugs before facing real ones, so it learns a project's quirks in advance
- 01It finds code regions actually exercised by the repository's own passing tests, has an LLM re-implement that functionality without seeing the original code (so it naturally makes plausible mistakes), and turns the resulting test failures into synthetic issues
- 02A SWE agent solves these synthetic issues, and both successful and failed attempts are mined into two kinds of knowledge: global 'diagnostic skills' (how to reason about and navigate a given code entity) and local 'intervention skills' (how to actually fix it and pitfalls to avoid), each tied to specific code entities
- 03When a real issue arrives, relevant global skills are retrieved upfront based on the issue description (using BM25 search), while local skills are injected on the fly the moment the agent touches the matching file or function
- 04Built on Mini-SWE-Agent and tested with DeepSeek-V3.2 and GPT-5-mini, SkillForge improved Pass@1 by +5.8 and +5.6 percentage points on SWE-bench Verified, and +5.8 and +4.1 points on SWE-bench Pro, outperforming all evaluated history-driven and online baselines
What they did
- It finds code regions actually exercised by the repository's own passing tests, has an LLM re-implement that functionality without seeing the original code (so it naturally makes plausible mistakes), and turns the resulting test failures into synthetic issues
- A SWE agent solves these synthetic issues, and both successful and failed attempts are mined into two kinds of knowledge: global 'diagnostic skills' (how to reason about and navigate a given code entity) and local 'intervention skills' (how to actually fix it and pitfalls to avoid), each tied to specific code entities
- When a real issue arrives, relevant global skills are retrieved upfront based on the issue description (using BM25 search), while local skills are injected on the fly the moment the agent touches the matching file or function
- Built on Mini-SWE-Agent and tested with DeepSeek-V3.2 and GPT-5-mini, SkillForge improved Pass@1 by +5.8 and +5.6 percentage points on SWE-bench Verified, and +5.8 and +4.1 points on SWE-bench Pro, outperforming all evaluated history-driven and online baselines
| Method | Model | Pass@1 | Avg Cost |
|---|---|---|---|
| Mini-SWE-Agent | DeepSeek-V3.2 | 66.4% | $0.049 |
| GPT-5-mini | 55.0% | $0.031 | |
| History-driven project-specific knowledge acquisition | |||
| SWE-Exp | DeepSeek-V3.2 | 69.0%†↑2.6% | $0.090 |
| GPT-5-mini | 56.6%†↑1.6% | $0.065 | |
| EvoCoder | DeepSeek-V3.2 | 67.0% ↑0.6% | $0.064 |
| GPT-5-mini | 58.4% ↑3.4% | $0.052 | |
| MemGovern | DeepSeek-V3.2 | 69.2%†↑2.8% | – |
| GPT-5-mini | 58.0%†↑3.0% | – | |
| Online project-specific knowledge acquisition | |||
| SAGE | DeepSeek-V3.2 | 67.2% ↑0.8% | $0.081 |
| GPT-5-mini | 56.0%†↑1.0% | $0.052 | |
| SWE-Debate | DeepSeek-V3.2 | 68.2% ↑1.8% | $0.382 |
| GPT-5-mini | 56.4% ↑1.4% | $0.167 | |
| Live-SWE-agent | DeepSeek-V3.2 | 67.0% ↑0.6% | $0.050 |
| GPT-5-mini | 55.6% ↑0.6% | $0.042 | |
| Variants of SkillForge | |||
| SkillForge w/ SWE-Smith | DeepSeek-V3.2 | 68.0% ↑1.6% | $0.088 |
| GPT-5-mini | 56.4%†↑1.4% | $0.071 | |
| SkillForge w/ LLM Summary | DeepSeek-V3.2 | 68.7%†↑2.3% | $0.069 |
| GPT-5-mini | 54.4% ↓0.6% | $0.065 | |
| SkillForge | DeepSeek-V3.2 | 72.2%†↑5.8% | $0.074 |
| GPT-5-mini | 60.6%†↑5.6% | $0.066 | |
| †: p−value<0.05. |
| DeepSeek-V3.2 | GPT-5-mini | |||
|---|---|---|---|---|
| Method | Pass@1 | Avg Cost | Pass@1 | Avg Cost |
| Mini-SWE-Agent | 28.3% | $0.047 | 47.6% | $0.063 |
| SWE-Exp | 29.4% ↑1.1% | $0.083 | 48.7% ↑0.9% | $0.089 |
| Live-SWE-agent | 32.4% ↑4.1% | $0.051 | 49.1%†↑1.5% | $0.072 |
| SkillForge | 34.1%†↑5.8% | $0.069 | 51.7%†↑4.1% | $0.087 |
| †: p−value<0.05. |
| Approach | DeepSeek-V3.2 | GPT-5-mini |
|---|---|---|
| w/o Global Diagnostic Skills | 68.4% (↓3.8%) | 57.6% (↓3.0%) |
| w/o Local Intervention Skills | 67.8% (↓4.4%) | 57.2% (↓3.4%) |
| SkillForge | 72.2% | 60.6% |
| Resolver LLM | Knowledge-source LLM | Pass@1 |
|---|---|---|
| GPT-5-mini | GPT-5-mini | 60.6% |
| DeepSeek-V3.2 | 55.0% | |
| DeepSeek-V3.2 | GPT-5-mini | 65.2% |
| DeepSeek-V3.2 | 72.2% |
Why it matters
When deploying coding agents on a real, specific codebase, the biggest obstacle is that the agent doesn't know the project's own conventions and traps yet -- this method offers a practical way to pre-train that knowledge even for projects with little or no historical issue data, without ballooning per-issue cost. It's directly relevant for teams building or deploying automated bug-fixing agents on their own repositories.
Terms in this paper
- Pass@1 · the percentage of issues correctly resolved on the very first attempt
- SWE-bench · a benchmark that evaluates AI coding agents on real GitHub issues
- BM25 · a classic text-retrieval algorithm that ranks documents by keyword overlap with a query
- entity-grounded · knowledge is tied directly to specific files, functions, or classes in the actual codebase rather than stored generically
- self-distillation · the process of extracting reusable knowledge from an agent's own generated experience rather than external data
Original abstract (English)
Large language model (LLM) based agents have demonstrated remarkable proficiency in automated software issue resolution, yet they often struggle to resolve issues in a specific repository because they lack project-specific knowledge. Existing self-evolving approaches acquire such knowledge from repository history or online repair trajectories, but they either depend on available historical issue-resolution signals or incur substantial per-issue test-time exploration cost. In this paper, we propose SkillForge, a self-distillation framework that proactively acquires project-specific knowledge from the repository itself. Instead of waiting for real issues to expose project-specific knowledge gaps, SkillForge synthesizes project-specific issues by re-implementing test-covered core functionalities of the repository. By resolving these synthetic issues, SkillForge distills reusable project-specific knowledge into entity-grounded skills and associates them with relevant repository entities for future issue resolution. Extensive experiments using both open-source and closed-source models show that SkillForge consistently improves issue resolution performance over strong baselines. These results demonstrate that proactively acquiring project-specific knowledge before solving real issues substantially improves downstream software issue resolution.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
Latest from METAL MEDIA
Figures: Silin Chen et al., arXiv:2608.18933, CC BY 4.0