在专门只要求删除的新基准CanItDelete的200个任务上,表现最好的模型(Claude Opus 4.8)成功率也只有79.0%;即便直接给出需要删除的精确代码行,GPT-5.6 Sol仍有19.5%的任务失败,原因是删除范围过大或又添加了新代码。给一个7B模型的训练数据中加入1.28万条删除样本(占训练token的0.7%)后,CanItDelete成功率提升13.9个百分点,SWE-bench Verified提升5.3分,CanItEdit提升1.4分。
Figure 1: An example of deletion avoidance in a solved SWE-bench Verified task. The developer patch removes a statement, while the model retains it under an ‘else’ guard.
Table 1: Mean deletion recall on 197 tasks all five models solve and 57 all five fail. δ denotes effect size.
Model
Failed (n=57)
Solved (n=197)
δ
GLM-4.6
24.0%
67.5%
0.532
GPT-5
29.9%
68.5%
0.485
Kimi-K2
19.8%
65.2%
0.543
Opus-4.5
30.4%
71.7%
0.504
Salesforce SAGE
27.5%
68.0%
0.501
Figure 2: Overview of CanItDelete benchmark construction, cumulative diagnostic modes, and the structural outcome taxonomy.
Table 2: Patch strategies among 2,358 classifier-labeled pairs. Share is overall frequency; pass rate is SWE-bench Verified resolution within each strategy. Table 12 (Appendix B) lists the ten Guard-and-Go forms.
Strategy
n
Share
Pass rate
Delete-and-Replace
1,197
50.8%
85.2%
Guard-and-Go
684
29.0%
72.2%
Non-reference alternative
477
20.2%
39.6%
Figure 3: Vanilla-mode success and failure composition across 12 models. Success is measured over 200 tasks; failure mechanisms are measured over each model’s failed tasks.
Table 3: Attempts passing the original suite and, among them, the deletion-sensitive check on 34 tasks per model. Drop is in percentage points (pp); each task is about 2.9 pp.
Passing attempts
Model
Tasks
Original suite
Original
suite
With deletion check
With deletion
check
Drop (pp)
Drop
(pp)
Original
suite
With deletion
check
Drop
(pp)
GPT-5.6 Sol
34
21 (61.8%)
15 (44.1%)
17.6
Opus 4.8
34
21 (61.8%)
14 (41.2%)
20.6
GLM-5.2
34
26 (76.5%)
18 (52.9%)
23.5
DeepSeek-V4-Pro
34
18 (52.9%)
10 (29.4%)
23.5
Overall
136
86 (63.2%)
57 (41.9%)
21.3
Figure 4: Diagnostic-ladder outcomes under increasingly precise deletion guidance. Four models use 200 tasks; Claude Opus 4.8 uses 173.
Table 4: 7B-model performance before and after deletion-augmented post-training. Values are three-run means (%); Δ is absolute percentage-point change. Over-deletion is a subset of complete-but-invalid edits.
CanItDelete outcome
Benchmark success
Succ.
Incomp.
Complete invalid
Over- deletion
SWE-b. Verified
EditBench
CanItEdit
Base
6.5
80.4
13.1
10.6
25.40
39.26
44.30
+Deletion
13.7
66.5
19.8
16.8
30.70
39.07
45.70
Δ
+7.2
−13.9
+6.7
+6.2
+5.30
-0.19
+1.40
Figure 5: File-, scope-, and exact-line overlap across all required deletions in the 197 tasks solved by all five models
Table 5: Model submissions selected from the official SWE-bench Verified repository. For each model family, we retained the latest submission using the OpenHands scaffold. The submission date is extracted from the leading YYYYMMDD component of the repository directory name.
Family
Selected model
Submission directory
Submission date
GLM
GLM-4.6
20250930_zai_ glm-4.6
September 30, 2025
GPT
GPT-5
20250807_ openhands_gpt5
August 7, 2025
Kimi
Kimi K2
20250716_ openhands_kimi_k2
July 16, 2025
Claude
Claude Opus 4.5
20251127_ openhands_ claude-opus-4-5
November 27, 2025
SAGE
Salesforce SAGE
20251103_ SalesforceAIResearch_ SAGE_OpenHands
November 3, 2025
Figure 6: Structural distributions across the 200 CanItDelete tasks.
Table 6: Mean per-task deletion precision and recall. A match requires the model-generated patch to delete the same base-commit source location as the official human patch.
Failed by all five (n=57)
Failed by all five
(n=57)
Solved by all five (n=197)
Solved by all five
(n=197)
Failed by all five
(n=57)
Solved by all five
(n=197)
Model
Precision
Recall
Precision
Recall
GLM-4.6
34.6%
24.0%
70.7%
67.5%
GPT-5
35.6%
29.9%
69.3%
68.5%
Kimi-K2
35.0%
19.8%
70.6%
65.2%
Opus-4.5
47.0%
30.4%
74.3%
71.7%
Salesforce SAGE
36.7%
27.5%
66.2%
68.0%
Table 7: Comparison of deletion recall between all-passed (n=197) and all-failed (n=57) eligible tasks for each evaluated model. Panel A reports the median and interquartile range (IQR). Panel B reports the Mann–Whitney U test with Holm-adjusted p-values and Cliff’s δ effect size.
All-passed
All-failed
Model
Median
IQR
Median
IQR
Salesforce SAGE
1.00
0.75
0.07
0.50
GLM-4.6
1.00
0.75
0.00
0.33
GPT-5
1.00
0.71
0.00
0.50
Kimi-K2
1.00
0.80
0.00
0.25
Opus-4.5
1.00
0.61
0.17
0.50
Table 8: Comparison of deletion precision between all-passed (n=197) and all-failed (n=57) eligible tasks for each evaluated model. Panel A reports the median and interquartile range (IQR). Panel B reports the Mann–Whitney U test with Holm-adjusted p-values and Cliff’s δ effect size.
All-passed
All-failed
Model
Median
IQR
Median
IQR
Salesforce SAGE
1.000
0.684
0.038
0.929
GLM-4.6
1.000
0.667
0.000
1.000
GPT-5
1.000
0.684
0.000
1.000
Kimi-K2
1.000
0.667
0.000
1.000
Opus-4.5
1.000
0.500
0.400
1.000
Table 9: Model-generated patch size relative to the corresponding developer patch for passing Guard-and-Go pairs. LOC is measured as the sum of added and deleted lines.
Model
Pairs
Model larger
Model
larger
Equal LOC
Equal
LOC
Model smaller
Model
smaller
Model
larger
Equal
LOC
Model
smaller
GLM-4.6
91
89 (97.80%)
0 (0.00%)
2 (2.20%)
GPT-5
120
58 (48.33%)
10 (8.33%)
52 (43.33%)
Kimi-K2
92
75 (81.52%)
3 (3.26%)
14 (15.22%)
Opus-4.5
91
30 (32.97%)
20 (21.98%)
41 (45.05%)
Salesforce SAGE
100
50 (50.00%)
14 (14.00%)
36 (36.00%)
Overall
494
302 (61.13%)
47 (9.51%)
145 (29.35%)
Table 10: Categories in the running taxonomy after each axial round. Each round covers five new pairs.
Round
Pairs coded
Added
Revised
Total
1
5
12
0
12
2
10
4
0
16
3
15
4
0
20
4
20
3
0
23
5
25
0
0
23
6
30
0
0
23
Table 11: Guard-and-Go structural subtypes identified in model-generated patches.
Subtype
Definition
Retained Path as Live Fallback
Guards the reported case but leaves logic deleted by the official patch as the default path, allowing non-matching inputs to continue executing the retained logic.
Special-Case Carve-Out Bypass
Adds a branch or relaxes a precondition for one input category while leaving the general resolution mechanism unchanged.
Missing-Element Existence Bypass
Checks for a required runtime element and returns a default value when it is absent instead of evaluating the retained logic.
Failure-Site Symptom Patch
Suppresses or compensates for the symptom at the failure site while leaving the producing control flow unchanged.
Exception Capture Bypass
Catches exceptions raised by retained logic and returns a default value instead of removing the underlying cause.
Post-Hoc Output Correction
Corrects an incorrect result after the buggy computation has already completed.
Missing-Functionality Injection
Introduces new logic or parameters to handle the reported case instead of adopting the restructuring performed by the official patch.
Upstream Reroute Around Retained Branch
Redirects problematic inputs away from a retained branch, leaving the obsolete branch as unreachable legacy logic.
Parallel Duplicate-Path Retention
Retains logic that the official patch consolidates while introducing an additional execution path that can also satisfy the behavior.
Value-Compatibility Shim
Conditionally restores a legacy constant or decoding behavior for a specific input.
Table 12: Distribution of the ten Guard-and-Go structural forms. Of the 684 Guard-and-Go pairs, the closed-source classifier assigned a subtype to 550 and abstained on the remaining 134. Share is computed over the 550 typed pairs, and pass rate is the proportion of pairs that SWE-bench Verified records as resolved.
Structural form
n
Share
Pass rate
Retained Path as Live Fallback
221
40.2%
68.3%
Special-Case Carve-Out Bypass
95
17.3%
52.6%
Missing-Element Existence Bypass
60
10.9%
81.7%
Failure-Site Symptom Patch
51
9.3%
66.7%
Exception Capture Bypass
43
7.8%
53.5%
Missing-Functionality Injection
35
6.4%
74.3%
Post-Hoc Output Correction
20
3.6%
80.0%
Upstream Reroute Around Retained Branch
10
1.8%
90.0%
Parallel Duplicate-Path Retention
8
1.5%
87.5%
Value-Compatibility Shim
7
1.3%
71.4%
All typed pairs
550
100.0%
67.3%
Table 13: Heuristic used to prioritize behaviorally important deletion units.
Signal
Score
Deleted lines are not reintroduced elsewhere
+3.0
Complete control block is removed
+3.0
Condition guarding existing code is changed or removed
+3.0
Control-flow statement is removed
+3.0
Complete function or class is removed
+2.5
Decorator is removed
+2.0
Comparison or Boolean logic is removed
+1.5
Function or class signature is changed
+1.5
Assignment is removed
+1.0
Function or method call is removed
+1.0
Deleted code contains problem-statement terms
+2.0
Larger multi-line deletion unit
+1.5
Import and its dependent code are removed
+0.5
Standalone import is removed
−4.0
Table 14: Task counts across the deletion-focused F2P construction process.
Selection stage
Count
Tasks with at least 25% deletion
69
Tasks included in deletion-unit analysis
65
Tasks with a meaningful code deletion
63
Tasks without a meaningful code deletion
2
Deletion units identified
166
Targets covered by a listed visible test
24
Targets not covered by a listed visible test
39
Host-validated deletion-focused F2P tasks
34
Table 15: The four cumulative CanItDelete diagnostic modes.
Mode
Cumulative signal
Diagnostic question
Vanilla
No signal beyond the developer-style request
Can the model infer and execute the complete deletion end to end?
Explicit deletion
Requires complete removal and forbids guards, comments, disabled branches, fallbacks, replacement code, and other workarounds
Is failure caused by not choosing a genuinely subtractive edit?
Region pointer
Identifies relevant functions, methods, or regions without revealing deletion boundaries
Is finding every required region the principal obstacle?
Exact lines
Supplies occurrence-specific spans and requires all unmarked text to be preserved
Can the model execute a precise deletion once intent, location, and boundaries are known?
Table 16: Complete five-model diagnostic-ladder results. Four models use 200 tasks per mode; Claude Opus 4.8 uses 173. Cells report count (percentage).
Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right fil
