SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

arXiv:2608.024992026-08-02

A benchmark that measures how badly coding agents stumble when a user actually edits the code mid-task

SWE-Touch is an evaluation framework that simulates a user directly modifying code while a coding agent is still working on a task, instead of just sending messages. It injects a plausible but task-conflicting fake user edit (Counter-Edit) near code the agent is actively inspecting, and measures how nine models respond on SWE-bench Verified and two longer-horizon benchmarks. Average resolve rate dropped by 7.7 percentage points, and being strong at solo coding did not guarantee stability under this kind of interruption.

METAL MEDIA explanatory visual

How SWE-Touch injects and evaluates user edits

Evidence statusMeasured results reported

Mine critical regionsOverlap the repair trajectories of three different models (GPT 5.5, GLM 5.1, MiniMax M2.7) to find code regions central to solving the task.
Generate & validate Counter-EditA separate User Patch Generator crafts a plausible but task-conflicting edit near those regions and checks it can't solve the task alone or combined with the reference fix.
Inject into shared workspaceWhenever the agent's actions touch the target region, the edit plus a contextual user message is applied to the live repository, up to three times by default.
Verify & classify failuresThe final code is checked against tests; failed cases are categorized into types such as keeping the conflicting code, replacing it wrongly, or only partially fixing it.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Existing coding-agent benchmarks mostly let agents work alone or only allow users to send messages, but the authors found that 59.0% of sessions in the released SWE-chat data actually contain repository edits attributed to the user.
SWE-Touch runs three different models (GPT 5.5, GLM 5.1, MiniMax M2.7) on each task and overlaps their trajectories to find task-critical code regions, then uses a separate User Patch Generator to construct a plausible edit near those regions that conflicts with completing the task.
Each candidate edit is validated with three checks: the fake user edit alone must not solve the task, the reference fix must solve it, and combining both must still fail — ensuring the edit is a genuine, non-trivial conflict.
During evaluation, whenever the agent's actions touch the target code region, the runtime injects this fake edit along with a contextual user message, up to three times per task by default, then lets the agent continue and checks the final result against tests.
Across nine models on SWE-bench Verified, average resolve rate dropped by 7.7 points under this Counter-Edit condition, and degradation persisted on the longer-horizon SWE-Bench Pro and DeepSWE benchmarks as well.

Figure 1: Users and agents share a workspace in real coding sessions. (a) A user edit alters the repository state that subsequent agent actions observe and modify. (b) Our analysis of the released SWE-chat data [5] finds that 59.0% of sessions contain repository changes attributed to the user.

Table 1: Comparison of user interaction modes in coding-agent benchmarks.
Benchmark	Code	User	User	Codebase
Repair	Simulator	Messages	Edit
SWE-bench Verified	✓	✗	✗	✗
Ambig-SWE	✓	✓	✓	✗
HiL-Bench	✓	✓	✓	✗
SWE-Interact	✓	✓	✓	✗
SWE-Together	✓	✓	✓	✗
SWE-Touch (ours)	✓	✓	✓	✓

Figure 2: Overview of SWE-Touch. Agent trajectories identify task-critical regions; a separate generator constructs and validates task-conflicting edits; evaluation injects each edit and its contextual message into the shared workspace before task verification.

Table 2: Average patch size, shown as changed lines / files.
Source	Reference repair	Counter-Edit
Lines / files	Lines / files
SWE-bench Verified	13.3 / 1.20	7.0 / 1.04
SWE-Bench Pro	361.0 / 5.44	13.0 / 1.40
DeepSWE	730.2 / 7.24	10.8 / 1.52

Figure 3: Longer-horizon robustness under user edits on SWE-Bench Pro and DeepSWE: (a) cost–performance shifts and (b) sensitivity to edit frequency.

Table 3: Main results on SWE-bench Verified over three runs. Resolve is mean ± standard deviation; Steps and Tok. (K) are completed-task means, with tokens reported in thousands. Δ is Counter-Edit minus Vanilla; Retention is the share of majority-solved Vanilla tasks that remain majority-solved under Counter-Edit. (1) indicates mean resolve-rate ranking; Rank Δ shows the corresponding ranking change.
	Vanilla	Counter-Edit	Δ	Retention	Rank
Model	Resolve	Steps	Tok. (K)	Resolve	Steps	Tok. (K)	Resolve	Steps	(%)	Δ
Claude Opus 4.8	85.2±1.8 (1)	24.5	367	83.3±0.6 (1)	30.9	506	-1.8	+6.4	96.0	–
GPT 5.5	80.5±1.0 (2)	33.2	1,344	79.2±0.6 (2)	31.3	1,141	-1.3	-1.9	95.0	–
GLM 5.1	72.7±2.0 (7)	55.0	1,007	68.3±0.8 (4)	64.5	1,272	-4.3	+9.5	83.3	↑3
MiniMax M2.7	76.5±1.5 (3)	45.3	865	62.7±2.4 (8)	47.7	904	-13.8	+2.4	78.1	↓5
MiniMax M2.5	75.7±3.3 (4)	45.1	863	66.2±1.0 (5)	47.5	908	-9.5	+2.4	78.3	↓1
Qwen 3.7 Max	75.2±1.0 (5)	29.9	420	70.3±0.8 (3)	31.1	424	-4.8	+1.2	90.3	↑2
Qwen3-Coder-480B	57.2±3.5 (9)	52.6	763	40.7±1.0 (9)	54.6	806	-16.5	+2.0	60.8	–
Kimi K2.6	70.3±2.0 (8)	62.5	1,381	64.3±3.4 (6)	62.1	1,296	-6.0	-0.4	87.2	↑2
DeepSeek V4 Pro	74.8±0.8 (6)	41.9	827	63.8±1.8 (7)	46.6	954	-11.0	+4.7	81.5	↓1

(b) Resolve rate under varying edit frequency (K) on two longer-horizon benchmarks.

Table 4: Mean resolve rates on SWE-Bench Pro and DeepSWE over two runs. Δ Res. is the change in resolve rate, and Δ Steps is the change in mean model calls per observed trajectory. Both differences are Counter-Edit minus Vanilla.
	SWE-Bench Pro	DeepSWE
Model	Vanilla	Counter-Edit	Δ Res.	Δ Steps	Vanilla	Counter-Edit	Δ Res.	Δ Steps
Claude Opus 4.8	68.0	68.0	0.0	+7.5	56.0	46.0	-10.0	+9.9
GPT 5.5	38.0	38.0	0.0	+0.9	64.0	56.0	-8.0	+4.9
GLM 5.1	43.1	32.8	-10.3	+19.1	19.4	16.8	-2.5	+31.4
MiniMax M2.7	30.6	24.6	-6.0	+9.4	2.2	2.2	0.0	+27.6
MiniMax M2.5	32.6	24.6	-8.0	+20.4	0.0	0.0	0.0	+33.1
Qwen 3.7 Max	36.0	26.0	-10.0	−1.4	4.1	2.1	-2.0	+2.3
Qwen3-Coder-480B	20.0	14.0	-6.0	+0.7	0.0	0.0	0.0	+7.8
Kimi K2.6	50.0	48.0	-2.0	−4.7	18.0	12.0	-6.0	+0.2
DeepSeek V4 Pro	34.0	32.0	-2.0	+8.9	4.1	2.0	-2.1	−7.6

Figure 4: Failure analysis of solved-to-unresolved runs across nine models on SWE-bench Verified. (a) Overall distribution across seven failure types. (b) Per-model composition. (c) Percentage of failures in which the agent revises or removes the user edit before termination. Sample sizes and audit details are reported in Appendix D.

Table 5: Resolve rates for message, code-edit, and edit-frequency ablations on SWE-bench Verified. Δ is relative to the paired Vanilla run.
Intervention	GPT 5.5	GLM 5.1	MiniMax M2.7	Qwen 3.7 Max
Resolve	Δ	Resolve	Δ	Resolve	Δ	Resolve	Δ
Vanilla	81.5	–	70.5	–	76.5	–	74.0	–
Message (K=3)	79.5	-2.0	73.0	+2.5	76.5	0.0	77.0	+3.0
Code edit (K=3)	80.5	-1.0	66.5	-4.0	67.0	-9.5	71.5	-2.5
Both (K=1)	78.5	-3.0	72.0	+1.5	64.5	-12.0	71.5	-2.5
Both (K=3)	79.5	-2.0	69.0	-1.5	64.5	-12.0	71.0	-3.0
Both (K=5)	78.0	-3.5	69.0	-1.5	60.0	-16.5	69.0	-5.0

Figure 5: Post-edit behavior on a diagnostic sample from SWE-bench Verified. (a) Mean read, edit, and test commands issued after the final user edit. (b) Agent response mode to the user edit.

Table 7: List prices used to convert tokens into dollars in Figure 3(a), as displayed by the serving endpoint on 27 July 2026; OpenRouter rates are the promotional prices shown at that time. Cached input is charged at the input rate.
Model	Endpoint	In ($/M)	Out ($/M)
Claude Opus 4.8	Anthropic	5.00	25.00
GPT 5.5	OpenAI	5.00	30.00
GLM 5.1	OpenRouter	0.966	3.036
MiniMax M2.7	OpenRouter	0.24	0.96
MiniMax M2.5	OpenRouter	0.15	0.90
Qwen 3.7 Max	OpenRouter	1.475	4.425
Qwen3-Coder-480B	OpenRouter	0.22	1.80
Kimi K2.6	OpenRouter	0.646	2.72
DeepSeek V4 Pro	OpenRouter	0.435	0.87

Figure 6: Task-level outcome transitions from Vanilla to Counter-Edit on SWE-bench Verified. Arrows indicate the direction and magnitude of shifts between solved and unresolved states.

Table 10: Verifier outcomes under Vanilla and Counter-Edit, by model.
	Van. solved	Van. unresolved
Model	kept	→unres.	→solved	kept
GPT 5.5	152	8	7	33
Claude 4.8	166	7	3	24
GLM 5.1	125	25	16	34
MiniMax M2.7	121	34	6	39
MiniMax M2.5	119	33	10	38
Qwen 3.7 Max	139	15	4	42
Qwen3-Coder	73	47	9	71
Kimi K2.6	123	18	9	50
DeepSeek V4	123	28	7	42

Findings

Averaged across nine models, resolve rate on SWE-bench Verified fell by 7.7 points under Counter-Edit versus solo (Vanilla) runs, with per-model losses ranging from 1.3 to 16.5 points.
The two strongest solo models, Claude Opus 4.8 (85.2%→83.3%) and GPT 5.5 (80.5%→79.2%), stayed nearly stable and kept their top ranks, while mid-tier models with similar Vanilla scores diverged sharply under Counter-Edit — e.g., MiniMax M2.7 fell from rank 3 to rank 8.
Auditing failed runs that were previously solved, 63.3% kept the conflicting user code active untouched, 13.9% replaced it with another wrong implementation, and 11.6% only partially reconciled the conflict.
Sending only a user message without changing the code had small, inconsistent effects (-2.0 to +3.0 points), whereas silently applying the code edit with no message caused a consistent decline across every model (-1.0 to -9.5 points).
A harmless control edit (Co-Edit) that didn't conflict with the task barely moved resolve rate on average (-0.1 points), indicating the core difficulty is the semantic conflict itself, not just an external modification appearing in the workspace.

Where it can be used

The framework could be used to stress-test coding assistants meant for pair-programming or live code review scenarios where a human might edit code while the agent is working.
It offers a way to check whether an agent habitually re-verifies and re-tests code after external changes, rather than assuming its own earlier plan is still valid.
Teams choosing among coding agents could use robustness under shared-workspace edits as a separate criterion, rather than relying solely on solo autonomous-benchmark scores.

Limits and open work

Results are based on specific sampled task sets (200 tasks from SWE-bench Verified, 25 each from SWE-Bench Pro and DeepSWE), so generalization to other codebases, languages, or task types is untested.
The simulated user messages were generated by GPT-4o rather than real humans, so they may not capture the full range of how actual users phrase or explain their edits.
Counter-Edits are deliberately constructed to conflict with the task in a controlled way; real user edits may not always be this clearly adversarial or well-defined.
On the longer-horizon benchmarks, edits are triggered by trajectory progress fractions rather than by region overlap, so those results aren't directly comparable to the main SWE-bench Verified setup.
The paper identifies needed capabilities — detecting changes, reconciling conflicts, and re-validating with targeted tests — but does not yet propose or test an agent design that reliably achieves all three.

Why it matters

If coding agents can't handle a human actually touching the shared codebase mid-task, that's a serious gap for real pair-programming and collaborative dev workflows, not just an academic curiosity. This work shows that leaderboard-style solo performance and robustness to shared-workspace collaboration are two different axes, which matters for anyone deciding which agent to trust in a live coding session.

Terms in this paper

Counter-Edit · A plausible but incorrect fake user code edit designed to conflict with completing the task
resolve rate · The percentage of tasks where all verification tests pass
retention · The share of tasks solved by majority vote under solo (Vanilla) runs that remain solved after the Counter-Edit is introduced
User Patch Generator · A separate agent whose only job is to construct and validate the task-conflicting fake user edits
shared workspace · A setting where the user and the agent both act on the same repository, files, and executable state

Original abstract (English)

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to

Authors · Yuqiao Tan

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Yuqiao Tan et al., arXiv:2608.02499, arxiv-nonexclusive