AI 비서가 업무를 한 번 성공했다고 믿을 수 있는 건 아니다 - 507개 업무 시나리오로 확인한 반복 신뢰성의 민낯

arXiv:2608.197412026-08-21

One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

마이크로소프트 연구팀이 AI 에이전트가 환불 처리, 예약 변경, 보험 청구 갱신 같은 실제 업무를 끝까지 제대로 해내는지 확인하는 테스트 환경 Thinkingbox와 507개 업무 벤치마크를 공개했다. 같은 작업을 20번 반복시켜 보니 가장 뛰어난 모델도 한 번이라도 성공하는 비율은 91.12%였지만, 20번 모두 성공하는 비율은 25.25%에 그쳤다. 실패한 시도의 상당수가 겉보기엔 깔끔하게 끝났고 도구 호출도 문제없어 보였지만, 실제로는 백엔드 상태가 틀려 있었다.

METAL MEDIA 해설 도표

AI 비서가 업무를 한 번 성공했다고 믿을 수 있는 건 아니다 - 507개 업무 시나리오로 확인한 반복 신뢰성의 민낯

01Thinkingbox는 AI 에이전트, 가상 사용자, 업무용 도구(주문 시스템, 예약 시스템 등)가 서로 대화하고 작업하는 과정을 격리된 환경에서 재현하고, 마지막에 데이터베이스 상태와 부작용까지 확인하는 테스트 도구다.
02이 위에서 소매, 여행/숙박, 자동차보험, 은행 내부 IT, 컨설팅 IT/HR 등 5개 분야에 걸친 507개의 업무 시나리오(Thinkingbox-bench)를 만들었고, 각 시나리오는 정답 하나가 아니라 여러 방법을 인정하되 잘못되거나 빠지거나 불필요한 결과는 걸러내는 실행 가능한 채점 기준을 갖췄다.
03GPT-5.2, GPT-5.4, o3-pro, Claude Sonnet/Opus 4.6, Grok-4.3 등 상용 모델과 DeepSeek-V4-Pro, GLM-5.1, Kimi-K2.6, Qwen 계열 등 공개 가중치 모델 12종을 각 작업당 20번씩 시험했다.
04가장 잘한 모델(GPT-5.4)도 한 번 시도했을 때 성공률(pass@1)은 65.36%였고, 20번을 모두 성공시켜야 인정하는 pass^20은 25.25%에 불과해, 우연히 성공하는 것과 매번 안정적으로 해내는 것 사이에 큰 격차가 있었다.
05실패 사례를 분석하니 전체 실패의 77.5%가 '도구 오류를 만나고도 복구하지 못함'이었고, 12.1%는 '도구 호출은 성공했지만 잘못된 결과로 상태를 바꿈'이었다. 실패했는데도 대화는 매끄럽게 끝나고 상태 변경 행동도 있어 보이는 경우가 많아, 응답 내용이나 도구 호출 여부만으로는 진짜 성공을 판단할 수 없었다.

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

Thinkingbox는 AI 에이전트, 가상 사용자, 업무용 도구(주문 시스템, 예약 시스템 등)가 서로 대화하고 작업하는 과정을 격리된 환경에서 재현하고, 마지막에 데이터베이스 상태와 부작용까지 확인하는 테스트 도구다.
이 위에서 소매, 여행/숙박, 자동차보험, 은행 내부 IT, 컨설팅 IT/HR 등 5개 분야에 걸친 507개의 업무 시나리오(Thinkingbox-bench)를 만들었고, 각 시나리오는 정답 하나가 아니라 여러 방법을 인정하되 잘못되거나 빠지거나 불필요한 결과는 걸러내는 실행 가능한 채점 기준을 갖췄다.
GPT-5.2, GPT-5.4, o3-pro, Claude Sonnet/Opus 4.6, Grok-4.3 등 상용 모델과 DeepSeek-V4-Pro, GLM-5.1, Kimi-K2.6, Qwen 계열 등 공개 가중치 모델 12종을 각 작업당 20번씩 시험했다.
가장 잘한 모델(GPT-5.4)도 한 번 시도했을 때 성공률(pass@1)은 65.36%였고, 20번을 모두 성공시켜야 인정하는 pass^20은 25.25%에 불과해, 우연히 성공하는 것과 매번 안정적으로 해내는 것 사이에 큰 격차가 있었다.
실패 사례를 분석하니 전체 실패의 77.5%가 '도구 오류를 만나고도 복구하지 못함'이었고, 12.1%는 '도구 호출은 성공했지만 잘못된 결과로 상태를 바꿈'이었다. 실패했는데도 대화는 매끄럽게 끝나고 상태 변경 행동도 있어 보이는 경우가 많아, 응답 내용이나 도구 호출 여부만으로는 진짜 성공을 판단할 수 없었다.

Figure 1: ThinkingBox overview and discovery–reliability gap. (A) ThinkingBox runs multi-turn interactions among a simulated user, an LLM agent, and isolated MCP-compatible tools, then evaluates terminal state, side effects, and dialogue with executable judges. More detailed pipeline can be found in Figure 2 (B) Across 507 tasks and 20 attempts per task, pass@20 is much higher than pass@1, while all-20 success is far lower, showing that successful trajectories are often discoverable but not reliably repeatable. Full results can be found in Table 4

Table 1: Comparison with representative agent and tool-use benchmarks. Thinkingbox-bench focuses on stateful business-domain workflows where the final verdict checks backend state, side effects, and dialogue outcome, while exposing tasks through MCP-compatible servers.
Benchmark	Primary domain	Tools/APIs	User dialogue	Stateful backend	Side-effect checks	MCP servers
SWE-bench (9)	Code repair	×	×	✓	×	×
BFCL (23)	Function calling	✓	×	×	×	×
ToolBench / API-Bank (25; 14)	API tool use	✓	×	×	×	×
WebArena / OSWorld (43; 36)	Web/desktop control	×	×	✓	×	×
AppWorld (30)	App APIs / coding agents	✓	×	✓	✓	×
MCP-Atlas (2)	Real MCP servers	✓	×	×	×	✓
τ-bench / τ2-bench (40; 3)	Domain APIs	✓	✓	✓	△	×
Thinkingbox-bench	Business tool workflows	✓	✓	✓	✓	✓

Figure 2: Overview of Thinkingbox. The sandbox orchestrates the interaction among a simulated user, LLM agents, isolated domain tools, side-effect extraction, and executable judges. The same trajectory-level verdict supports benchmark evaluation, failure analysis, and agent training.

Table 2: Key statistics for the ThinkingBox-Bench domains.
	Retail	Booking	Insurance	Neobank	Consulting
Tasks	98	104	100	104	101
Backend systems	11	8	7	3	18
Databases (tables / rows)	22 / 86	17 / 98	14 / 72	20 / 151	30 / 231
Agent tools (write / read)	16 / 17	10 / 28	14 / 19	13 / 19	13 / 14
Policy (words)	945	3,684	2,471	3,392	1,747
Knowledge base (documents)	9	11	8	8	9
Actions per task	4.4 (1–10)	8.8 (4–19)	4.7 (1–11)	6.7 (3–12)	5.8 (1–13)
Evaluation	DB state	DB + rubrics	DB state	DB + rubrics	DB state

Figure 3: Pass@k progression of models used for evaluation

Table 3: Representative Thinkingbox-bench scenario patterns. Each scenario is instantiated as a concrete task with an initial backend state, available MCP-compatible tools, and executable checks.
Domain	Scenario	Required agent behavior	Executable checks
Retail / e-commerce	User asks to change or refund part of an order.	Identify the correct order/item, verify eligibility, request missing confirmation, and update only the relevant record.	Correct order state; refund/order side effect; no unrelated customer or item modified.
Travel / hospitality	User requests a booking change under date, room, or policy constraints.	Check reservation, availability, and change policy before modifying booking or explaining denial.	Correct reservation state; price/fee side effect if applicable; policy-compliant dialogue.
Auto insurance	User reports or updates a claim.	Verify policy and vehicle/incident details, collect missing information, and create or update the claim.	Correct ticket/claim state; no coverage mutation unless allowed.
Neobank support	User asks to dispute, freeze, or modify an account/card action.	Authenticate relevant account context, distinguish reversible and irreversible actions, and apply only valid updates.	Correct account/card state; required dispute/freeze side effect; no unrelated account changed.
Consulting IT/HR	Employee asks for access, HR, or internal support changes.	Verify role, approval, or employee record, then create ticket or update access according to policy.	Correct ticket/access state; approval and provisioning side effects; no unrelated records modified

Figure 4: pass^k progression of models used for evaluation

Table 4: Thinkingbox-bench pass@1 (%) by domain. Each task is evaluated with N=20 repeated trials, and scores are micro-averaged over trials and tasks. Size is reported as total/activated parameters for MoE models. The o3-pro row excludes 636 system/harness-error trials; its valid denominators therefore vary by domain. Task-cluster bootstrap intervals are reported in Appendix D.1.
Model	Size	Retail (98)	Auto (100)	Booking (104)	Bank (104)	Consulting (101)	Average
Proprietary models
GPT-5.4	–	76.33	62.65	68.13	65.34	54.60	65.36
GPT-5.2	–	70.20	22.40	53.70	51.15	34.06	46.28
o3-pro	–	37.94	2.96	24.16	24.37	14.75	20.60
Claude Sonnet 4.6	–	68.93	58.20	60.38	53.99	51.14	58.45
Claude Opus 4.6	–	74.90	14.65	28.89	38.03	34.21	37.91
Grok-4.3	–	43.93	2.60	15.14	1.78	9.55	14.38
Open-weights models
DeepSeek-V4-Pro	1.6T/49B	68.21	29.65	43.13	44.86	31.04	43.26
GLM-5.1	744B/40B	58.67	25.70	35.43	13.27	34.06	33.19
Kimi-K2.6	1T/32B	53.72	24.50	39.52	33.65	37.33	37.66
Mistral-Large-3	675B/41B	11.28	1.30	8.99	1.15	0.74	4.66
Qwen3.6-27B	27B	43.11	29.00	46.39	27.84	18.37	32.94
Qwen3.5-9B	9B	19.15	0.45	4.52	1.06	2.34	5.41

Figure 5: Domain failure-rate heatmap (%) on Thinkingbox-bench. Each cell is 100−pass@1 within the domain; darker red indicates higher failure rate.

Table 5: Failure mode breakdown (%) on Thinkingbox-bench. For each model, values indicate the percentage distribution of dominant failure types across failed trials.
Model	Tool Usage	No State-Changing Action	Incomplete User Resolution	Wrong State Update
GPT-5.4	89.6	1.6	0.5	8.3
Claude Sonnet 4.6	84.0	2.8	4.2	8.9
GPT-5.2	80.1	0.7	0.9	18.3
DeepSeek-V4-Pro	69.8	0.8	24.7	4.7
Claude Opus 4.6	78.1	3.0	10.2	8.7
Kimi-K2.6	85.2	1.3	12.1	1.4
GLM-5.1	88.1	1.3	3.5	7.0
Qwen3.6-27B	75.0	0.6	13.1	11.2
o3-pro	67.4	3.9	0.9	27.8
Grok-4.3	69.2	6.8	1.5	22.5
Mistral-Large-3	65.8	4.6	15.2	14.4
Average	77.5	2.5	7.9	12.1

Table 6: Composition of the 507-task evaluation set. Every case has executable backend-state checks. “Backend only” cases use no additional response rubric, whereas “Backend + rubric” cases also impose a binary final-response requirement.
Domain	Backend only	Backend + rubric	Task families represented in the final set
Retail / e-commerce	98	0	Delivery delay and exception handling; delivered-but-missing and return-to-sender cases; returns, refunds, exchanges, and warranty claims; installation scheduling and cancellation; order cancellation, promotions, and membership changes.
Travel / hospitality	89	15	Individual, corporate, and group booking changes; payment recovery; cancellations and refunds; corporate invoices and account benefits; group billing and services; hotel-partner verification and discrepancies; special requests and post-stay complaints.
Auto insurance	100	0	Billing extensions and arrangements; proof-of-insurance documents; adding, removing, or updating drivers and vehicles; first notice of loss and claim intake; listed-driver requests; reinstatement and policy cancellation.
Neobank internal IT support	89	15	Employee access and approval requests; password and account-security actions; production-incident access; hardware troubleshooting, assignment, replacement, and procurement; software and license requests; policy-information questions.
Consulting IT / HR support	101	0	Client-system and document access; software provisioning; expenses; hardware requests; employee onboarding; training enrollment; engagement and approval checks; corporate-travel policy and escalation.
Total	477	30	507 executable cases in total.

Table 7: Artifacts in a final Thinkingbox-bench case and the corresponding construction or review question.
Artifact	Contents	Review question
User goal g	Initial request plus facts the simulated user can provide during follow-up	Is the request natural, internally consistent, and resolvable without access to hidden evaluator information?
Initial state b0	Synthetic records in the domain backend, including existing tickets and related business objects	Do all referenced identifiers resolve, and do cross-system records agree before the agent acts?
Policy context	Domain operating manual and the fixed evaluation time	Does the policy determine eligibility, approvals, disclosures, and allowed actions without exposing the golden result?
Tools 𝒯	MCP-compatible read and write operations over the isolated domain services	Can the required evidence be retrieved and the intended outcome be executed using available tools?
User simulation 𝒰	Task-specific user role used for on-policy follow-up dialogue	Does the user provide only task-consistent facts and allow necessary clarification?
Checks 𝒞	Expected backend state and, for designated cases, final-response requirements	Does the evaluator accept the intended outcome and reject missing, wrong, or extra effects?

Table 8: Representative evaluator targets drawn from the final task families. These are outcome conditions, not prescribed action sequences.
Case type	Required outcome	Incorrect effects rejected
Retail return or delivery exception	Correct order, return/refund/replacement, and support-ticket state under the applicable policy	Wrong order or item, ineligible refund, duplicate ticket, incorrect ticket status, or missing compensation record
Booking modification or cancellation	Correct booking dates, room/board attributes, charges or refund, and ticket or hotel escalation when required	Modification without availability or policy support, wrong fee, partial group update, or confidential partner information disclosed
Insurance billing, policy, or claim request	Correct policy-linked extension, driver/vehicle change, claim, document, cancellation, or reinstatement state	Identity or eligibility bypass, wrong effective date, unintended coverage change, or incorrect ticket resolution
Internal access, hardware, or software request	Correct employee, approval, access, asset, procurement, notification, and ticket records	Excess privilege, bypassed approval, wrong assignee or device, duplicate request, or incomplete multi-system update
Consulting operations request	Correct engagement-linked access, expense, onboarding, training, hardware, or travel outcome	Missing prerequisite, incorrect approval path, inconsistent cross-system records, or premature ticket closure

Table 9: Azure API inference parameters for the agent, fixed GPT-5.4-mini simulated user, and fixed GPT-5.4-mini response judge.
Parameter	Agent	Simulated User	Response Judge
temperature	1.0	0.3	0.0
max_completion_tokens	4096	4096	128
is_reasoning	True	False	False
reasoning_effort	medium	none	none
top-p	1.0	1.0	1.0
frequency penalty	0.0	0.0	0.0
number of completions	1	1	1
timeout (seconds)	600	600	600

Table 10: vLLM serving parameters for the locally hosted Qwen-series models.
Parameter	Value
port	8000
data parallel size	8
tensor parallel size	1
maximum model length	65,536
reasoning parser	qwen3
automatic tool choice	True
tool-call parser	qwen3_coder
API interface	Chat Completions
API endpoint	/v1/chat/completions

Table 11: Repeated-trial discovery and reliability on Thinkingbox-bench. CI denotes a 95% task-cluster bootstrap interval. pass^20 requires all 20 attempts to pass, while pass@20 requires at least one. 0/20 and 20/20 means the number of tasks that have never been passed, and has always passed during all 20 attempts. We include Qwen3-8B (38) and MiniMax-M2.5 (18).
Model	pass@1 [95% CI]	pass^20	pass@20	0/20 tasks	20/20 tasks
GPT-5.4	65.36% [62.23%, 68.51%]	25.25%	91.12%	45	128
Claude Sonnet 4.6	58.45% [55.23%, 61.66%]	20.12%	88.56%	58	102
GPT-5.2	46.28% [43.19%, 49.39%]	8.68%	84.81%	77	44
DeepSeek-V4-Pro	43.26% [40.27%, 46.20%]	3.55%	84.62%	78	18
Claude Opus 4.6	37.91% [34.48%, 41.40%]	13.81%	70.02%	152	70
Kimi-K2.6	37.66% [35.02%, 40.30%]	3.16%	84.22%	80	16
GLM-5.1	33.19% [30.23%, 36.14%]	2.76%	69.82%	153	14
Qwen3.6-27B	32.94% [30.20%, 35.75%]	2.37%	78.50%	109	12
o3-pro	20.60% [18.27%, 23.02%]	1.65%	61.54%	195	4
Grok-4.3	14.38% [12.34%, 16.50%]	0.00%	45.96%	274	0
Qwen3.5-9B	5.84% [5.39%, 6.31%]	0.03%	30.77%	351	0
Mistral-Large-3	4.66% [3.66%, 5.75%]	0.00%	25.44%	378	0
MiniMax-M2.5	0.19% [0.00%, 0.53%]	0.00%	0.59%	504	0
Qwen3-8B	0.13% [0.01%,0.34%]	0.00%	0.99%	502	0

Table 12: Retrospective evaluator ablation study. The first block counts failed trials that nevertheless appear complete under weaker response- or action-level evaluators. The second block reports evidence exposed by our executable database comparison.
Signal among executable-check failures	Trials	Share of failures
Failed trials accepted by weak observable evaluators
Clean termination	67,763	84.86%
Clean termination + state-changing tool call	64,586	80.88%
Above + no explicit error in final tool response	53,697	67.24%
Evidence reported by executable state/side-effect checks (ours)
Database hash mismatch	79,015	98.95%
Wrong field value	61,973	77.61%
Collection-length mismatch	45,510	56.99%
Missing expected state or side effect	20,250	25.36%
Extra unintended state or side effect	34,575	43.30%

Table 13: Failure distribution (%) by domain on Thinkingbox-bench. Each row reports percentages over failed trials in that domain.
Domain	Tool Usage	No State-Changing Action	Incomplete User Resolution	Wrong State Update
Retail	70.7	4.3	12.9	12.1
Travel	81.2	1.0	13.3	4.5
Auto insurance	48.2	3.8	15.1	32.9
Neobank internal IT	79.0	6.0	8.4	6.6
Consulting IT/HR	71.4	5.4	10.2	13.0

Table 14: Average trajectory-level interaction counts on Thinkingbox-bench. All columns are mean counts per trial, computed from parsed traces. Write calls are tool calls whose names indicate state-changing actions such as create, update, cancel, refund, or modify.
Model	Avg. msgs. / trial	Avg. tool calls / trial	Avg. write calls / trial	Avg. tool errors / trial
GPT-5.4	29.80	11.05	3.91	1.98
Claude Sonnet 4.6	34.41	10.16	3.62	1.68
GPT-5.2	36.40	11.71	4.74	2.10
DeepSeek-V4-Pro	38.94	11.68	3.78	2.34
Claude Opus 4.6	31.20	9.99	3.21	1.57
Kimi-K2.6	35.62	11.70	3.60	2.91
GLM-5.1	44.86	11.99	3.91	2.58
Qwen3.6-27B	28.40	11.11	3.82	1.68
o3-pro	30.58	7.52	2.78	1.35
Grok-4.3	24.15	8.74	3.13	2.06
Mistral-Large-3	25.72	9.22	3.15	1.81
MiniMax-M2.5	25.50	5.56	1.63	0.99
Qwen3.5-9B	31.31	11.75	3.61	0.85
Qwen3-8B	19.76	5.29	2.07	0.66

Table 15: Average token usage per model turn and average model turns per trial by model and domain. Each cell reports “Tokens/Turn x Turns”, with tokens in thousands; both averages use trials with token metadata. Tokens include input context and output tokens for each model invocation, with cached input counted once.
Model	Retail	Travel	Auto insurance	Neobank internal IT	Consulting IT/HR
GPT-5.4	14.2k × 8.1	27.4k × 7.2	14.6k × 9.6	21.3k × 6.7	18.2k × 7.4
Claude Sonnet 4.6	14.4k × 8.3	29.8k × 7.3	16.7k × 8.9	25.2k × 7.5	16.9k × 7.7
GPT-5.2	12.9k × 12.7	30.0k × 9.0	14.2k × 12.9	21.6k × 8.2	17.8k × 16.8
DeepSeek-V4-Pro	19.8k × 8.4	31.9k × 8.0	21.3k × 9.7	25.8k × 7.9	24.3k × 8.6
Claude Opus 4.6	14.9k × 8.7	27.3k × 7.6	15.8k × 10.1	25.1k × 7.0	17.1k × 7.6
Kimi-K2.6	13.1k × 10.0	23.3k × 8.6	13.6k × 11.9	19.2k × 9.0	17.3k × 10.2
GLM-5.1	17.7k × 10.0	28.5k × 9.8	20.1k × 12.3	25.2k × 10.6	22.5k × 11.6
Qwen3.6-27B	18.5k × 13.6	27.2k × 13.3	19.9k × 13.5	24.0k × 11.8	22.0k × 16.2
o3-pro	12.3k × 10.1	24.0k × 10.1	14.5k × 10.2	18.8k × 9.2	15.6k × 11.9
Grok-4.3	16.6k × 10.9	28.7k × 9.7	18.4k × 9.6	24.8k × 9.8	21.3k × 15.8
Mistral-Large-3	16.0k × 12.5	30.3k × 13.1	18.1k × 12.6	23.2k × 9.6	19.6k × 10.8
MiniMax-M2.5	17.0k × 8.8	16.4k × 6.2	13.9k × 5.7	14.9k × 6.0	17.1k × 7.2
Qwen3.5-9B	17.7k × 13.5	29.1k × 14.53	19.9k × 14.2	24.0k × 13.7	22.9k × 18.3
Qwen3-8B	15.7k × 8.87	23.3k × 11.6	18.7k × 8.9	20.8k × 8.29	19.9k × 9.11

왜 중요한가

고객센터 자동화나 사내 IT 지원처럼 실제 시스템 상태를 바꾸는 AI 에이전트를 도입하려는 기업이라면, 한두 번의 데모 성공만으로 신뢰해서는 안 된다는 근거를 제공한다. 도구 호출이 문법적으로 맞고 대화가 자연스럽게 끝났다고 해서 실제 업무가 올바르게 처리됐다고 볼 수 없다는 점을 구체적 수치로 보여준다.

이 논문의 용어

MCP · AI 에이전트가 외부 도구·서비스를 표준화된 방식으로 호출하도록 만든 연결 규격
pass@k · 같은 작업을 k번 시도했을 때 적어도 한 번 성공할 확률
pass^k · 같은 작업을 k번 시도했을 때 전부 성공해야 인정하는 엄격한 성공률
POMDP · 에이전트가 전체 상황을 다 볼 수 없는 상태에서 순차적으로 결정을 내리는 상황을 수학적으로 표현한 모델
부작용(side effect) · 작업 도중 의도치 않게 발생한 추가적인 데이터베이스 변경이나 결과

저자 · Zhuochun Li, Youngmin Ko, Ali Keramati, Nicola Ferri, Susana Palmaz Lopez Pelaez, Liang-Chun Tsai, Calvin Wang, Mirco Mi

arXiv에서 원문 보기

METAL MEDIA 최신 기사

그림 출처: Zhuochun Li et al., arXiv:2608.19741, CC BY 4.0

AI 비서가 업무를 한 번 성공했다고 믿을 수 있는 건 아니다 - 507개 업무 시나리오로 확인한 반복 신뢰성의 민낯

AI 비서가 업무를 한 번 성공했다고 믿을 수 있는 건 아니다 - 507개 업무 시나리오로 확인한 반복 신뢰성의 민낯

무엇을 했나

왜 중요한가

이 논문의 용어

최신 논문

METAL MEDIA 최신 기사