컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

AI 코딩 에이전트의 '정답률'은 명령어가 실행 직전에 다시 파싱될 때 무너진다

arXiv:2608.135472026-08-12

QuoteBench: How Matched Scores Can Hide Command-Path Failures

AI 코딩 에이전트의 '정답률'은 명령어가 실행 직전에 다시 파싱될 때 무너진다

AI 코딩 에이전트가 생성한 Bash 명령어는 실행되기 전에 한 번 더 감싸지거나 재해석되는 경우가 많은데, 기존 벤치마크는 이 지점에서 벌어지는 손상과 모델 자체의 생성 오류를 구분하지 못했다. QuoteBench는 56개 과제로 이 둘을 실험적으로 분리해, 명령어가 큰따옴표 안에 다시 삽입되는 상황(중첩 전송)에서 성공률이 55.4~73.2점(퍼센트 포인트) 떨어지고, 이 위험을 모델에 미리 알려주면 6개 설정에서 30.4~60.7점이 회복됨을 보였다. GPT-5.6-sol처럼 겉보기 점수 변화는 -3.6점에 불과하지만 실제로는 -64.3점의 손상과 +60.7점의 보완이 서로 상쇄된 경우도 발견됐다.

METAL MEDIA 해설 도표

AI 코딩 에이전트의 '정답률'은 명령어가 실행 직전에 다시 파싱될 때 무너진다

  1. 01문제의식: LLM 에이전트가 만든 Bash 명령어는 원격 접속(ssh), 컨테이너 실행, CI 파이프라인 등을 거치며 큰따옴표 안에 다시 끼워 넣어지는데, 이 과정에서 따옴표나 특수문자가 깨져도 기존 평가 방식(정답 여부만 보는 매칭 점수)으로는 원인을 알 수 없었다.
  2. 02방법: 저자는 14개 실제 장애 사례에서 뽑은 14개 작업군, 56개 과제로 QuoteBench를 만들고, 모델이 명령어를 만드는 방식(생성 계약)과 그 명령어가 실행되는 경로(실행 전송)를 독립적으로 교차시켰다. 같은 모델 응답을 원본 그대로 실행할 때와, 일부러 이스케이프 처리를 하지 않은 파서를 하나 추가해 재실행할 때를 비교해 손상분을 측정하고, 모델에게 '네 답은 큰따옴표 안에 다시 넣어질 것'이라고 미리 알려주는 조건(공개된 경계)을 추가해 회복분을 측정했다.
  3. 03결과: 원본 그대로 실행하면 최상위 모델들은 이미 91.1~100%로 거의 포화 상태였지만, 재파싱 과정을 하나 추가하는 것만으로 성공률이 55.4~73.2점 떨어졌다. 이 위험을 미리 고지하면 8개 설정 중 6개에서 30.4~60.7점이 회복됐고, 나머지 2개는 거의 변화가 없거나 소폭 나빠졌다.
  4. 04의미 있는 발견: 겉보기 점수(매칭 점수)만 보면 GPT-5.6-sol은 거의 변화가 없어 보이지만(-3.6점), 실제로는 큰 손상(-64.3점)과 큰 회복(+60.7점)이 우연히 상쇄된 결과였다. 또한 실행 경로를 바꾸는 것만으로 26개 모델 비교 쌍 중 최소 1개는 순위가 명백히 뒤집혔다.
  5. 05실용적 시사점: 올바른 이스케이프 처리나 임시 스크립트 사용 같은 간단한 해법으로 이 문제를 완전히 없앨 수 있지만, 실제 에이전트 시스템 6곳을 조사한 결과 이런 경계 처리가 모델이 아니라 그 뒤 단계(하네스)에서 이뤄지고 있어 모델 혼자서는 통제할 수 없는 경우가 많았다.
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 문제의식: LLM 에이전트가 만든 Bash 명령어는 원격 접속(ssh), 컨테이너 실행, CI 파이프라인 등을 거치며 큰따옴표 안에 다시 끼워 넣어지는데, 이 과정에서 따옴표나 특수문자가 깨져도 기존 평가 방식(정답 여부만 보는 매칭 점수)으로는 원인을 알 수 없었다.
  2. 방법: 저자는 14개 실제 장애 사례에서 뽑은 14개 작업군, 56개 과제로 QuoteBench를 만들고, 모델이 명령어를 만드는 방식(생성 계약)과 그 명령어가 실행되는 경로(실행 전송)를 독립적으로 교차시켰다. 같은 모델 응답을 원본 그대로 실행할 때와, 일부러 이스케이프 처리를 하지 않은 파서를 하나 추가해 재실행할 때를 비교해 손상분을 측정하고, 모델에게 '네 답은 큰따옴표 안에 다시 넣어질 것'이라고 미리 알려주는 조건(공개된 경계)을 추가해 회복분을 측정했다.
  3. 결과: 원본 그대로 실행하면 최상위 모델들은 이미 91.1~100%로 거의 포화 상태였지만, 재파싱 과정을 하나 추가하는 것만으로 성공률이 55.4~73.2점 떨어졌다. 이 위험을 미리 고지하면 8개 설정 중 6개에서 30.4~60.7점이 회복됐고, 나머지 2개는 거의 변화가 없거나 소폭 나빠졌다.
  4. 의미 있는 발견: 겉보기 점수(매칭 점수)만 보면 GPT-5.6-sol은 거의 변화가 없어 보이지만(-3.6점), 실제로는 큰 손상(-64.3점)과 큰 회복(+60.7점)이 우연히 상쇄된 결과였다. 또한 실행 경로를 바꾸는 것만으로 26개 모델 비교 쌍 중 최소 1개는 순위가 명백히 뒤집혔다.
  5. 실용적 시사점: 올바른 이스케이프 처리나 임시 스크립트 사용 같은 간단한 해법으로 이 문제를 완전히 없앨 수 있지만, 실제 에이전트 시스템 6곳을 조사한 결과 이런 경계 처리가 모델이 아니라 그 뒤 단계(하네스)에서 이뤄지고 있어 모델 혼자서는 통제할 수 없는 경우가 많았다.
Figure 1: Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract-then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The R​R and N​N cells are matched. The R​N cell measures fixed-reply damage, and N​N−R​N is the realized contract-conditioned contrast. Panel (b) works the decomposition for GPT-5.6-sol.
Figure 1: Crossing generation contract with execution transport. Task, model configuration, and validator are fixed across cells. The reply is fixed within each transport replay pair. Cell labels are contract-then-transport. On the generation axis, R denotes the raw contract and N the disclosed-boundary contract. On the transport axis, R denotes raw execution and N the nested transport. The R​R and N​N cells are matched. The R​N cell measures fixed-reply damage, and N​N−R​N is the realized contract-conditioned contrast. Panel (b) works the decomposition for GPT-5.6-sol.
Table 1: One worked QuoteBench task (write-file, tier 3). The ⟨⟨⋅⟩⟩ markers delimit the exact literal payload and are not part of it. The ↩ symbol marks a newline that the payload itself contains. The validator compares exact final bytes, so any command that reaches the target state passes. Appendix Table 7 maps the full family set to the covered mechanisms. The artifact contains all 56 instructions, fixtures, oracles, and validators.
InstructionCreate a file named out.txt whose content is exactly the text ⟨⟨back\slash and `date` here↩ ’single’ "double" $(pwd) 100%⟩⟩ followed by a single trailing newline.
Hazardsbackslash, backtick, command substitution, multiline, mixed quotes
Naive commandecho "back\slash and `date` here …" > out.txt; the backticks execute date, $(pwd) expands locally, and the payload’s inner double quote terminates the string early
Final-state gateexact bytes of out.txt, including one trailing newline, with no collateral files
One passing commanda single-quoted printf with an embedded-quote splice (the machine oracle). Any other correct encoding receives equal credit
Figure 2: Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix Table 13 lists the queried effort settings and request parameters. Table 2 separately summarizes the best observed setting per base model. Columns follow the mechanism groups in Appendix Table 7. SSH-like tasks use the local two-shell simulation. Rows are ordered by matched-nested score in this frozen sweep, which differs from the best-observed order in Table 2.
Figure 2: Matched-nested success varies by operation family across the frozen configuration sweep. Each cell is the percentage of a family’s four tasks passed by one stored generation. Rows retain the 15 frozen configurations used for broad coverage; Qwen rows marked (thinking) enable the think toggle and unmarked Qwen rows are non-thinking, and Appendix Table 13 lists the queried effort settings and request parameters. Table 2 separately summarizes the best observed setting per base model. Columns follow the mechanism groups in Appendix Table 7. SSH-like tasks use the local two-shell simulation. Rows are ordered by matched-nested score in this frozen sweep, which differs from the best-observed order in Table 2.
Table 2: Best-observed QuoteBench scorecard. Each base model contributes the measured setting with the highest matched-nested All-56 score. Ties prefer default, then lower mean provider-reported output tokens. default means that the request omitted the effort field. The number and names of measured settings differ by provider and appear in Figure 4 and Appendix B. This table is descriptive: each cell is one stored trial-0 generation and the selection is a within-model maximum over rungs; Appendix E.1.6 reports draw-to-draw spread. The Qwen rows expose only a think toggle rather than an effort ladder, so their best-observed setting is taken from the same-window sweep; for Qwen3.5-27B, the only Qwen size with a ladder row in Table 12, the resulting one-task difference is serving-window drift. Fixed same-window configurations support the mechanism analysis in Table 4.
ModelBest observed settingControlHostileAll 56 (%)Hostile LOFO range (%)
GPT-5.5xhigh14/1442/42100.0[100.0, 100.0]
Opus-5xhigh14/1442/42100.0[100.0, 100.0]
Fable-5max14/1442/42100.0[100.0, 100.0]
GPT-5.6-solhigh14/1441/4298.2[97.4, 100.0]
Opus-4.8max12/1441/4294.6[97.4, 100.0]
Gemini-3.1-Prolow14/1437/4291.1[87.2, 89.7]
Sonnet-4.6high9/1427/4264.3[61.5, 69.2]
Gemini-3.5-Flashmedium10/1426/4264.3[59.0, 66.7]
Haiku-4.5medium8/1413/4237.5[28.2, 33.3]
Qwen3.5-27Bnon-think8/149/4230.4[15.4, 23.1]
Qwen3.5-4Bthink5/147/4221.4[10.3, 17.9]
Qwen3.5-9Bnon-think5/145/4217.9[5.1, 12.8]
Gemini-3.1-Flash-Litedefault6/142/4214.3[2.6, 5.1]
Figure 3: Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The R​R value is raw success, R​N is the nested-replay pass rate of raw-conditioned replies, and N​N is matched nested success. The N​N−R​N gap is compensation. Appendix Table 14 reports every rung, including the byte-identical Gemini-3.1-Flash-Lite settings.
Figure 3: Where effort raises matched success, the nested-replay pass rate changes little. Each point uses 56 tasks, trial 0, and the GNU replay. The R​R value is raw success, R​N is the nested-replay pass rate of raw-conditioned replies, and N​N is matched nested success. The N​N−R​N gap is compensation. Appendix Table 14 reports every rung, including the byte-identical Gemini-3.1-Flash-Lite settings.
Table 3: Absolute GNU replay pass rates in the separate Study-B campaign, pooled over each model’s measured effort ladder and three trials per cell.
ModelAttemptsRaw (%)Native (%)Δ (pp)
Opus-4.884095.498.0+2.6
Opus-584098.297.4−0.8
Fable-584099.397.1−2.1
Gemini-3.1-Pro50498.895.0−3.8
GPT-5.6-sol67296.994.3−2.5
Gemini-3.5-Flash67295.785.7−10.0
Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched-nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order. Backward or dominated segments show measured non-monotonicity. Panels group model families for legibility, and Qwen sizes contribute two-point think-toggle trajectories.
Figure 4: Effort settings are provider-specific operating points. Each point shows trial-0 matched-nested success on the 56-task core against mean provider-reported output tokens per task, including reported hidden reasoning. Dashed segments follow the provider’s declared order. Backward or dominated segments show measured non-monotonicity. Panels group model families for legibility, and Qwen sizes contribute two-point think-toggle trajectories.
Table 4: All four crossover cells for the eight same-window configurations. Cell notation follows Figure 1: generation contract precedes transport. Cells are pass rates (%). Effects are percentage points. Damage is R​N−R​R, compensation is N​N−R​N, and the matched gap is N​N−R​R. Appendix Table 14 reports the interaction at every measured rung, and Table 15 gives the sensitivity tests.
ModelR​RR​NN​RN​NDamageComp.Matched gap
GPT-5.6-sol94.630.455.491.1−64.3+60.7−3.6
GPT-5.5100.028.650.089.3−71.4+60.7−10.7
Opus-596.430.442.989.3−66.1+58.9−7.1
Gemini-3.1-Pro98.225.033.980.4−73.2+55.4−17.9
Gemini-3.5-Flash96.428.667.958.9−67.9+30.4−37.5
Opus-4.891.126.862.557.1−64.3+30.4−33.9
Qwen3.5-27B85.730.483.930.4−55.40.0−55.4
Gemini-3.1-Flash-Lite78.619.680.414.3−58.9−5.4−64.3
Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered scale. Blank cells mark rungs the provider does not expose. Models follow descending GNU replay campaign effect. Labels are not comparable compute budgets across providers.
Figure 5: Native-minus-raw pass-rate change by provider effort rung, in percentage points on a zero-centered scale. Blank cells mark rungs the provider does not expose. Models follow descending GNU replay campaign effect. Labels are not comparable compute budgets across providers.
Table 5: Private-v2 crossover on 42 hostile payloads under the single-clause disclosed-boundary contract. The private set is hostile-only and not difficulty-matched to the public core, so absolute rates are interpreted within this set. Cells are pass rates. Effects are percentage points. Damage is R​N−R​R, compensation is N​N−R​N, interaction is (N​N−N​R)−(R​N−R​R), and the matched gap is N​N−R​R.
ModelR​RR​NN​RN​NDamageComp.InteractionMatched gap
GPT-5.6-sol92.919.050.097.6−73.8+78.6+121.4+4.8
Opus-4.892.916.759.542.9−76.2+26.2+59.5−50.0
Table 6: Private-v1 fixed raw replies under three transports on 42 tasks. This campaign predates the private-v2 crossover in Table 5. The two tables therefore use different generations. The first three columns report tasks passed out of 42. Script gain is the temporary-script rate minus the nested-wrapper rate, in percentage points.
ModelRawNested wrapperTemporary scriptScript gain (pp)
GPT-5.6-sol41/428/4241/42+78.6
Opus-4.840/427/4240/42+78.6
Qwen3.5-27B30/429/4230/42+50.0
Table 7: Mechanism groups used to construct the frozen core; the mapping documents coverage of the surveyed failure mechanisms. Benchmark analyses weight the 14 operation families equally.
Mechanism groupRepresentative failuresQuoteBench families
Literal quote and expansionApostrophes, double quotes, dollars, backticks, multiline payloadswrite-file, JSON writing, Git commit, environment passing, heredoc writing
Word splitting and path semanticsSpaces, globs, leading dashes, hostile filenames, argument boundariesargv passing, hostile filenames, find/glob, bulk rename
Embedded-language escapingRegex versus literal matching, sed replacement, AWK string processinggrep count, sed replace, field lookup, JSON writing
Second parser or remote-like expansionLocal expansion before a second shell, argument joining, heredoc transportSSH-like nested execution, SSH-like heredoc
Command-boundary representationCommand string, shell stdin, temporary file, argv, provider tool schemaraw/nested crossover, native-tool study, script bypass, typed pilot
Table 8: Command boundaries in six public agent systems, inspected at fixed commits. Contract is what the system asks the model to produce. A nested boundary arises on a separate axis from what the command targets. Observed boundary is what the harness then does with the reply. The classification concerns the parser boundary only. Sources: 20; 27; 13; 28; 22; 18.
SystemContractSource anchorObserved boundary
Codexnativecore/src/shell.rs:20--30, commit fa1d4c4command string becomes shell -c/-lc R
SWE-agentrawagents.py:936--967 and swe_env.py:197--222, commit 3ea751cagent action enters a persistent Bash session
LangChainnativeshell_tool.py:217--232,491--515, commit b3a6d9astructured command: string is written to shell stdin
Terminal-Benchrawtmux_session.py:26--33,75--173, commit d28711dcommand/key strings enter an interactive shell through tmux
OpenHandsnativeacp-command.ts:11--45,65--147, commit 850bd64human-readable command is tokenized to argv, and spawn has no shell
AutoGennative_docker_code_executor.py:327--363, commit 027ecf0generated code is written to a temporary file and invoked by argv
Table 9: Real-ssh grounding. Each stored raw reply is replayed through bash -c and through a real ssh localhost "R" remote command; ssh damage is the second minus the first. It matches the synthetic nested damage to the decimal for seven of eight configurations; the disclosed-reply cells are grounded in Appendix Table 21. Zero model calls.
ConfigurationRawsshssh damageNested damage
GPT-5.6-sol94.630.4−64.3−64.3
GPT-5.5100.028.6−71.4−71.4
Opus-596.430.4−66.1−66.1
Gemini-3.1-Pro98.226.8−71.4−73.2
Gemini-3.5-Flash96.428.6−67.9−67.9
Opus-4.891.126.8−64.3−64.3
Qwen3.5-27B85.730.4−55.4−55.4
Gemini-3.1-Flash-Lite78.619.6−58.9−58.9
Table 10: Campaign map. All replays are zero-call executions of stored replies in the pinned container.
CampaignGenerationsDesignFeeds
Study A same-window sweep8 configs × 56 × 2 contractsone randomized window, effort unsetTables 4, 15
Study A ladder sweep44 rungs, 11 configsper-provider windowsTables 2, 12
Study A rung crossover30 rungs, 7 configs, 26 crossover ptsreplay both transportsTable 14, Fig. 3
Public three-draw repeat8 configs × 56 × 2 × 2 drawssame design as the sweepAppendix E.1.6
Study B native tool8,736 generations, 17,472 replaysobservational, both userlandsTables 3, 17, 18
Private-v2 crossover2 models × 42 hostile payloadsone serving windowTable 5
Private-v1 replay3 models × 42 tasksearlier generations; script bypassTable 6
BSD-live legacy6 configurationsBSD-elicited, GNU-replayedTable 16
Real-ssh grounding8 configs × 56ssh localhost replayTable 9
Advice arm8 configs × 56 × 2same-window paired advice contrastTable 19
Grammar crossover8 configs × 56 × 2 disclosedreplay-only wrapper 2×2Table 20
Real-ssh full crossover6 configs × 56 × 2disclosed replies on real sshTable 21
JSON boundary6 configs × 56serializer replayTable 22
Table 11: Calibration of the unset-effort sweep arm against each configuration’s labelled ladder. Distance is unset success minus the lowest-rung success, in percentage points. The two measurements come from different serving windows, so small differences are descriptive only.
ModelUnset (%)Nearest rungDistance to lowest rung
GPT-5.6-sol91.1low0.0
GPT-5.589.3high−5.4
Opus-589.3medium+28.6
Gemini-3.1-Pro80.4none within the ladder−10.7
Gemini-3.5-Flash58.9low+1.8
Opus-4.857.1xhigh+17.9
Gemini-3.1-Flash-Lite14.3all four rungs tie0.0
Table 12: Matched-nested effort ladders under the disclosed-boundary contract. Within each row, success rates and mean provider-reported output tokens follow the setting order in the second column. Each point uses trial 0 over 56 tasks.
ModelSettings (in order)Success (%)Mean output tokens
GPT-5.6-sollow / medium / high / xhigh91.1 / 92.9 / 98.2 / 98.2362 / 565 / 773 / 1,073
GPT-5.5low / medium / high / xhigh94.6 / 92.9 / 91.1 / 100.0507 / 655 / 1,164 / 2,757
Opus-5low / medium / high / xhigh / max60.7 / 91.1 / 96.4 / 100.0 / 98.2796 / 1,336 / 1,458 / 2,421 / 3,499
Fable-5low / medium / high / xhigh / max71.4 / 75.0 / 96.4 / 92.9 / 100.0332 / 569 / 843 / 1,212 / 2,396
Opus-4.8low / medium / high / xhigh / max39.3 / 48.2 / 50.0 / 62.5 / 94.6835 / 1,291 / 1,706 / 3,481 / 6,960
Gemini-3.1-Prolow / medium / high91.1 / 91.1 / 91.15,267 / 5,753 / 6,308
Sonnet-4.6low / medium / high / xhigh23.2 / 28.6 / 64.3 / 62.528 / 1,411 / 4,337 / 4,539
Gemini-3.5-Flashminimal / low / medium / high57.1 / 58.9 / 64.3 / 62.53,851 / 3,108 / 3,081 / 3,389
Haiku-4.5low / medium / high / xhigh32.1 / 37.5 / 32.1 / 26.85,432 / 4,717 / 4,708 / 4,783
Qwen3.5-27Bnon-thinking / thinking28.6 / 28.620 / 7,489
Gemini-3.1-Flash-Liteminimal / low / medium / high14.3 / 14.3 / 14.3 / 14.320 / 20 / 20 / 20
Table 13: Study-A model identifiers and request parameters. The effort column lists exactly the settings queried; sweep arms omitted the effort field. Temperature is reported only where the interface accepts it.
Display nameModel identifierEffort settings queriedDecoding parameters
GPT-5.6-solgpt-5.6-sollow, medium, high, xhighmax output tokens 16,000, temperature not sent
GPT-5.5gpt-5.5low, medium, high, xhighmax output tokens 16,000, temperature not sent
Opus-5claude-opus-5low, medium, high, xhigh, maxprovider defaults, no sampling or length control sent
Opus-4.8claude-opus-4-8low, medium, high, xhigh, maxprovider defaults, no sampling or length control sent
Fable-5claude-fable-5low, medium, high, xhigh, maxprovider defaults, no sampling or length control sent
Sonnet-4.6claude-sonnet-4-6low, medium, high, xhighprovider defaults, no sampling or length control sent
Haiku-4.5claude-haiku-4-5low, medium, high, xhighprovider defaults, no sampling or length control sent
Gemini-3.1-Progemini-3.1-pro-previewlow, medium, hightemperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.5-Flashgemini-3.5-flashminimal, low, medium, hightemperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Gemini-3.1-Flash-Litegemini-3.1-flash-lite-previewminimal, low, medium, hightemperature 0, max tokens 4,096 in the sweep, omitted in the ladder
Qwen3.5-27BQwen/Qwen3.5-27Bnon-thinking, thinkingtemperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-9BQwen/Qwen3.5-9Bnon-thinking, thinkingtemperature 0, max tokens 4,096 non-thinking, omitted thinking
Qwen3.5-4BQwen/Qwen3.5-4Bnon-thinking, thinkingtemperature 0, max tokens 4,096 non-thinking, omitted thinking
Table 14: Crossover at every measured rung. Cells are percentages and effects are percentage points, defined as in Table 4. An asterisk marks the descriptive masked-fragility rule. Gemini-3.1-Flash-Lite returns byte-identical replies at all four settings and is retained only to document that the provider did not expose a usable ladder.
ConfigurationRungR​RR​NN​RN​NDamageCompensationMatched gap
GPT-5.6-sollow98.228.653.691.1−69.6+62.5−7.1
GPT-5.6-solmedium94.630.448.292.9−64.3+62.5−1.8∗
GPT-5.6-solhigh98.232.148.298.2−66.1+66.1+0.0∗
GPT-5.6-solxhigh98.230.451.898.2−67.9+67.9+0.0∗
Opus-5low98.228.657.160.7−69.6+32.1−37.5
Opus-5medium100.032.141.191.1−67.9+58.9−8.9
Opus-5high96.430.442.996.4−66.1+66.1+0.0∗
Opus-5xhigh98.232.142.9100.0−66.1+67.9+1.8∗
Opus-5max100.032.146.498.2−67.9+66.1−1.8∗
Gemini-3.1-Prolow100.025.032.191.1−75.0+66.1−8.9
Gemini-3.1-Promedium98.226.841.191.1−71.4+64.3−7.1
Gemini-3.1-Prohigh98.226.835.791.1−71.4+64.3−7.1
Fable-5low100.030.448.271.4−69.6+41.1−28.6
Fable-5medium100.030.448.275.0−69.6+44.6−25.0
Fable-5high98.232.142.996.4−66.1+64.3−1.8∗
Fable-5xhigh100.033.946.492.9−66.1+58.9−7.1
Fable-5max100.030.435.7100.0−69.6+69.6+0.0∗
Gemini-3.5-Flashminimal96.423.275.057.1−73.2+33.9−39.3
Gemini-3.5-Flashlow92.925.067.958.9−67.9+33.9−33.9
Gemini-3.5-Flashmedium96.423.276.864.3−73.2+41.1−32.1
Gemini-3.5-Flashhigh96.423.271.462.5−73.2+39.3−33.9
Opus-4.8low87.528.669.639.3−58.9+10.7−48.2
Opus-4.8medium92.928.667.948.2−64.3+19.6−44.6
Opus-4.8high96.433.964.350.0−62.5+16.1−46.4
Opus-4.8xhigh98.232.160.762.5−66.1+30.4−35.7
Opus-4.8max98.230.444.694.6−67.9+64.3−3.6∗
Gemini-3.1-Flash-Liteminimal78.619.680.414.3−58.9−5.4−64.3
Gemini-3.1-Flash-Litelow78.619.680.414.3−58.9−5.4−64.3
Gemini-3.1-Flash-Litemedium78.619.680.414.3−58.9−5.4−64.3
Gemini-3.1-Flash-Litehigh78.619.680.414.3−58.9−5.4−64.3
Table 15: Enumerated and Holm-adjusted two-sided family-sign p values for the two primary Study-A components under the single-clause disclosed-boundary contract. Effect sizes are percentage points.
Fixed-reply transportContract-conditioned compensation
ModelEffect [95% CI]Enum. pHolm pEffect [95% CI]Enum. pHolm p
GPT-5.6-sol−64.3 [−80.4,−46.4].000244.001465+60.7 [+46.4,+75.0].000244.001953
GPT-5.5−71.4 [−85.7,−55.4].000244.001465+60.7 [+44.6,+75.0].000244.001953
Opus-5−66.1 [−82.1,−50.0].000244.001465+58.9 [+41.1,+75.0].000488.002930
Gemini-3.1-Pro−73.2 [−87.5,−58.9].000122.000977+55.4 [+33.9,+73.2].001221.006104
Gemini-3.5-Flash−67.9 [−82.1,−51.8].000244.001465+30.4 [+12.5,+48.2].013672.041016
Opus-4.8−64.3 [−80.4,−46.4].000244.001465+30.4 [+14.3,+48.2].003906.015625
Qwen3.5-27B−55.4 [−69.6,−41.1].000244.0014650.0 [0.0,0.0]1.0000001.000000
Gemini-3.1-Flash-Lite−58.9 [−71.4,−48.2].000122.000977−5.4 [−10.7,0.0].250000.500000
Table 16: Cross-userland transfer in the earlier BSD-live campaign. Each cell reports pass rate on BSD live execution and GNU replay of the identical stored command. Arms retain their original campaign multiplicity: raw cells contain 56 records, while nested cells contain 56 or 168 depending on the configuration. We interpret only the within-arm BSD→GNU change. The final column summarizes the higher-transfer userland; it is not a counterfactual GNU-targeted generation result.
ModelRaw BSD→GNUNested BSD→GNUTransfers better to
Fable-5→91.1→87.5BSD
Qwen3.5-27B (non-thinking)→87.5→32.1GNU
Qwen3.5-27B (thinking)→82.1→50.0GNU
Gemini-3.1-Pro→100.0→96.4GNU
Gemini-3.5-Flash→96.4→67.9BSD
Opus-4.8→87.5→73.8BSD (raw); tie (nested)
Table 17: Exploratory native-minus-raw effects over each measured effort ladder, ordered by GNU replay effect.
BSD liveGNU replay
ModelΔLOFOPF/FPΔLOFOPF/FP
Opus-4.8+3.10[+2.18,+3.46]5/31+2.62[+1.67,+2.95]11/33
Opus-5−0.36[−0.51,0.00]12/9−0.83[−1.03,−0.51]16/9
Fable-5+0.95[0.00,+1.15]3/11−2.14[−2.44,−0.64]22/4
GPT-5.6-sol−1.19[−1.92,−0.48]25/17−2.53[−3.37,−0.64]26/9
Gemini-3.1-Pro+1.98[−0.64,+3.21]8/18−3.77[−4.91,−0.64]23/4
Gemini-3.5-Flash−4.02[−4.33,−3.04]37/10−9.97[−10.74,−6.89]77/10
Table 18: Study-B execution outcomes, aggregated across six models. Each row partitions 4,368 executions. Adherence denotes an invalid one-call tool invocation. Syntax includes parser and command-usage errors. Exit-0 wrong is a silent final-state failure.
UserlandContractPassAdherenceSyntaxNonzeroExit-0 wrong
BSDraw42300315948
BSDnative42362094162
GNUraw42520393641
GNUnative4146202212852
Table 19: Matched nested success (of 56) under the disclosed contract without and with escaping advice, paired within one serving window. Advice barely moves the top of the ladder, moves the middle of the ladder the most, and does not reach the bottom.
ConfigurationAdvice-freeWith adviceΔ (pp)
GPT-5.6-sol5449−8.9
GPT-5.54953+7.1
Opus-55254+3.6
Sonnet-4.63246+25.0
Haiku-4.52532+12.5
Opus-4.83034+7.1
Qwen3.5-27B1718+1.8
Gemini-3.1-Flash-Lite78+1.8
Table 20: Grammar crossover (tasks passed of 56). Rows are the disclosed grammar, columns the executed wrapper; the diagonal is matched, the anti-diagonal mismatched. The diagonal advantage separates the same top, middle, and bottom groups as the matched-nested scores.
Double disclosedSingle disclosedDiag−anti
Configurationon doubleon singleon singleon double(pp)
GPT-5.6-sol5475310+80.4
GPT-5.5498548+77.7
Opus-55284516+65.2
Sonnet-4.632112316+25.0
Opus-4.830112318+21.4
Haiku-4.525132011+18.8
Qwen3.5-27B17111117+0.0
Gemini-3.1-Flash-Lite719919−19.6
Table 21: Real-ssh full crossover (tasks passed of 56). Damage is R​N−R​R and compensation is N​N−R​N, both on the real ssh path. The final column repeats the synthetic nested compensation for comparison.
ConfigurationRRRNNRNNssh comp.Nested comp.
GPT-5.6-sol53173151+60.7+60.7
GPT-5.556162850+60.7+60.7
Opus-554172450+58.9+58.9
Opus-4.851153532+30.4+30.4
Qwen3.5-27B48174717+0.0+0.0
Gemini-3.1-Flash-Lite44114510−1.8−5.4
Table 22: JSON serializer boundary (tasks passed of 56). Correct-serializer damage is ≈0; naive-embedding damage is comparable to the shell nested transport. Unparseable counts replies whose naive JSON embedding fails to parse.
ConfigurationRawCorrectNaiveNaive damage
GPT-5.6-sol535321−57.1
GPT-5.5565619−66.1
Opus-5545422−57.1
Opus-4.8515118−58.9
Qwen3.5-27B484819−51.8
Gemini-3.1-Flash-Lite44448−64.3

왜 중요한가

에이전트가 실제로 서버에 접속하거나 컨테이너를 다루는 배포 환경에서는 이 논문이 지적한 '재파싱 경계'가 항상 존재하며, 이를 무시한 벤치마크 순위는 실제 배포 성능과 다를 수 있다. 코딩 에이전트를 도입하려는 개발자나 평가자는 모델 점수 하나만 보지 말고 어떤 실행 경로와 계약 조건에서 측정됐는지 함께 확인해야 한다.

이 논문의 용어

  • 매칭 점수(matched score) · 모델이 만든 명령어를 곧바로 실행해 얻는, 일반적으로 보고되는 성공률
  • 생성 계약(generation contract) · 모델에게 명령어를 어떤 형식·전제로 만들라고 지시하는 방식
  • 실행 전송(execution transport) · 모델이 만든 명령어가 실제 셸에 도달하기까지 거치는 경로, 예를 들어 그대로 실행되거나 다시 따옴표로 감싸져 실행되는 방식
  • 중첩 전송(nested transport) · 명령어가 ssh나 컨테이너 실행처럼 큰따옴표 안에 한 번 더 삽입되어 재해석되는 실행 경로
  • 최종 상태 검증(final-state validation) · 명령어 실행 후 파일, 인자, 깃 기록 등 결과 상태가 정확히 맞는지만 확인하는 채점 방식

저자 · Shangao Li

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: Shangao Li et al., arXiv:2608.13547, CC BY 4.0