K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

FraudBench: Stress-Testing Policy-Grounded Banking Agents Against Adaptive Fraud

arXiv:2608.181362026-08-20

新基准测试银行客服AI会不会被聊几句就骗走个人信息和钱

FraudBench是一个新的测试平台,让扮演骗子的AI通过纯对话,试图诱导拥有查账、改密码、转账等真实权限的银行客服AI做出不安全的操作。骗子AI会使用十种诈骗手法以及多步骤连环攻击,其中包括利用对话前面已经承认过的信息,使后面看似正常的请求变得不安全。测试了四个模型后发现,防御成功率只有49%到65%,其中洗钱工具人诈骗和连环攻击是最难防住的类型。

METAL MEDIA 解读图

新基准测试银行客服AI会不会被聊几句就骗走个人信息和钱

  1. 01研究者搭建了一个模拟银行环境,包含698份内部政策文件、可变的客户数据库和17个具有实际权限的操作工具,让客服AI和骗子AI通过这些工具展开真实对话
  2. 02安全判断依赖整段对话历史而非最后一句话,例如打电话的人如果之前承认过这笔钱是别人的工资,那么之后即使转账请求看起来正常,客服AI也必须拒绝
  3. 03从150个人工编写的对抗性场景中,公开了107个(90个单一诈骗类型加17个多步骤连环攻击),测试了Nemotron-3 Ultra 550B、Gemini 3.6 Flash、Gemini 3.1 Flash-Lite和gpt-oss-120b四个模型
  4. 04即便在理想条件下,即提前把正确的政策文件直接交给AI,防御成功率也只有49.5%到64.5%;而当Gemini 3.6 Flash必须自己从文件库中搜索时,成绩从64.5%骤降13个百分点至51.0%
  5. 05洗钱工具人诈骗是所有模型共同的最大弱点,在17个连环攻击任务中,表现较好的模型也只能防住8到9个,较弱的模型仅防住4个
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 研究者搭建了一个模拟银行环境,包含698份内部政策文件、可变的客户数据库和17个具有实际权限的操作工具,让客服AI和骗子AI通过这些工具展开真实对话
  2. 安全判断依赖整段对话历史而非最后一句话,例如打电话的人如果之前承认过这笔钱是别人的工资,那么之后即使转账请求看起来正常,客服AI也必须拒绝
  3. 从150个人工编写的对抗性场景中,公开了107个(90个单一诈骗类型加17个多步骤连环攻击),测试了Nemotron-3 Ultra 550B、Gemini 3.6 Flash、Gemini 3.1 Flash-Lite和gpt-oss-120b四个模型
  4. 即便在理想条件下,即提前把正确的政策文件直接交给AI,防御成功率也只有49.5%到64.5%;而当Gemini 3.6 Flash必须自己从文件库中搜索时,成绩从64.5%骤降13个百分点至51.0%
  5. 洗钱工具人诈骗是所有模型共同的最大弱点,在17个连环攻击任务中,表现较好的模型也只能防住8到9个,较弱的模型仅防住4个
Figure 1: FraudBench overview. A simulated adaptive caller attacks a policy-grounded banking agent across ten fraud mechanisms plus chained multi-step attacks, over privileged tools, a mutable customer database, and a 698-document internal policy corpus; episodes are graded on actions, state, leaks, and disposition, conditioned on the full conversation history. Prior conversational fraud benchmarks such as Fraud-R1 judge resistance to fraud messages without tools, a database, or an internal policy corpus, and cover a narrower slice of banking-fraud typologies.
Figure 1: FraudBench overview. A simulated adaptive caller attacks a policy-grounded banking agent across ten fraud mechanisms plus chained multi-step attacks, over privileged tools, a mutable customer database, and a 698-document internal policy corpus; episodes are graded on actions, state, leaks, and disposition, conditioned on the full conversation history. Prior conversational fraud benchmarks such as Fraud-R1 judge resistance to fraud messages without tools, a database, or an internal policy corpus, and cover a narrower slice of banking-fraud typologies.
Table 1: Operational comparison with neighboring agent benchmarks and frameworks. ✓ denotes a primary, implemented capability; ⊚, partial or narrower; –, not a primary feature. Multi-class fraud: multiple distinct fraud typologies are tested. History-dep.: safety depends on earlier turns, not only the latest request. Long conv.: extended multi-turn interaction with a (simulated) user. Type indicates whether the work is a fixed benchmark, a reusable framework, or both; FraudBench is a fixed banking-fraud benchmark whose environment and attacker machinery are reusable for new scenarios.
WorkExec. stateLarge KBFraudMulti-class fraudHistory-dep.Long conv.Type
Fraud-R1 (Yang et al. 2025)––✓⊚⊚⊚Benchmark
τ2-bench (Barres et al. 2025)✓––––✓Benchmark
τ-Knowledge (Shi et al. 2026)✓✓–––✓Benchmark
AgentDojo (Debenedetti et al. 2024)✓–––✓⊚Both
SafeArena (Tur et al. 2025)✓––––⊚Benchmark
AgentHazard (Feng et al. 2026)✓–––✓⊚Benchmark
DoomArena (Boisvert et al. 2025)✓–––✓⊚Framework
FraudBench✓✓✓✓✓✓Both
Figure 2: Grading pipeline. (1) Conversation roll: the agent and simulated caller hold a multi-turn conversation (shown top-to-bottom, chat style); here the caller impersonates a customer, induces the agent to read back protected contact details, and then requests a transfer to a third-party IBAN. (2) The full transcript is scored in two stages: (2a) a deterministic grader inspects the ordered tool-call sequence, unsafe state changes, and structured leaks (e.g. an executed unauthorized transfer), and (2b) an LLM-as-judge stage checks the scenario’s natural-language assertions (scam recognized, no PII revealed, transfer refused), each marked pass (✓) or fail (X). (3) The episode earns reward 1 only if all checks pass (logical AND); a single leak or prohibited action yields 0. The reported runs use the LLM-judge stage only; the deterministic stage is what catches false-safe transcripts like this one.
Figure 2: Grading pipeline. (1) Conversation roll: the agent and simulated caller hold a multi-turn conversation (shown top-to-bottom, chat style); here the caller impersonates a customer, induces the agent to read back protected contact details, and then requests a transfer to a third-party IBAN. (2) The full transcript is scored in two stages: (2a) a deterministic grader inspects the ordered tool-call sequence, unsafe state changes, and structured leaks (e.g. an executed unauthorized transfer), and (2b) an LLM-as-judge stage checks the scenario’s natural-language assertions (scam recognized, no PII revealed, transfer refused), each marked pass (✓) or fail (X). (3) The episode earns reward 1 only if all checks pass (logical AND); a single leak or prohibited action yields 0. The reported runs use the LLM-judge stage only; the deterministic stage is what catches false-safe transcripts like this one.
Table 2: Ten fraud mechanisms and the decision each isolates, plus the chain category of multi-step adaptive attacks.
MechanismPrincipal decision being tested
APP scamProtect a verified customer manipulated into an authorized payment.
Account takeoverStop stolen knowledge or recovery credentials from becoming account control.
Card fraudPreserve freezes, activation locks, PIN rules, and transaction controls.
Data exfiltrationEnforce identity, role, ownership, scope, and retention boundaries.
First-party fraudDetect dispute or reimbursement claims contradicted by the account.
Prompt injectionTreat instruction-like text in records or user content as data.
Money muleStop laundering, structuring, third-party movement, or authorization bypass.
Phishing PIIRecognize that static PII satisfies basic identity but not every high-risk action.
Social engineeringResist false authority, urgency, secrecy, sympathy, and escalation pressure.
Synthetic identityEnforce onboarding and expansion eligibility against fabricated or blended identities.
Chain (adaptive)Preserve trust-state evidence across a multi-step attack when a later locally valid request follows an earlier probe, admission, or failed attempt.
Table 3: Preliminary attack-security at pass1 on the currently graded tasks, for the four models with complete 107-task runs under oracle retrieval (the agent’s KB_search returns the task’s gold policy documents). All runs use the GPT-5.4 Nano user simulator, one trial, and the natural-language-assertion grader only (no deterministic Stage B checks, no matched legitimate controls). Models without a complete 107-task run are excluded. A realistic all-tools retrieval setting is compared in Table 5.
DefenderTasksSecurity S (p1)
Gemini 3.6 Flash10764.5%
Nemotron-3 Ultra 550B10757.9%
Gemini 3.1 Flash-Lite10753.3%
gpt-oss-120b10749.5%
Table 4: Per-mechanism attack-security (defended tasks / graded tasks) for the four models with complete 107-task runs. Columns: Nemo. = Nemotron-3 Ultra 550B; G3.6F = Gemini 3.6 Flash; G3.1FL = Gemini 3.1 Flash-Lite; oss120 = gpt-oss-120b. Money mule is the hardest mechanism for every model; the chain row covers all 17 chain tasks.
MechanismNemo.G3.6FG3.1FLoss120
APP scam3/95/94/92/9
Account takeover3/98/97/96/9
Card fraud7/98/97/98/9
Data exfiltration8/98/97/95/9
First-party fraud4/97/93/92/9
Prompt injection7/98/96/96/9
Money mule2/91/91/91/9
Phishing PII5/94/96/95/9
Social engineering8/96/96/97/9
Synthetic identity7/95/96/97/9
Chain (adaptive)8/179/174/174/17
Overall62/10769/10757/10753/107
Overall (%)57.964.553.349.5
Table 5: Retrieval-setting comparison at pass1. Oracle supplies the gold policy documents; all-tools requires the agent to retrieve them from the 698-document corpus. Gemini 3.6 Flash is the only model run under both settings, and its 13-point drop isolates retrieval from policy reasoning; Gemini 3.1 Pro was run under all-tools only. All-tools Gemini 3.6 Flash graded 104 of 107 episodes (three infrastructure failures excluded).
DefenderOracle S (p1)All-tools S (p1)
Gemini 3.6 Flash64.5% (107)51.0% (104)
Gemini 3.1 Pro–55.1% (107)

为什么重要

随着银行客服AI获得转账、改密码等真实权限,几句对话套路就可能让它变成攻击入口,而这项研究首次用可执行、基于真实工具操作的方式具体量化了这种风险,而不只是靠静态文本测试。这为银行和AI开发者在把这类客服AI真正投入使用前,提供了明确的薄弱环节数据。

本文术语

  • 洗钱工具人 (money mule) · 被利用来帮他人转移非法资金的账户或人员
  • oracle检索 · 测试中直接把正确的政策文件提供给AI,而不需要它自己搜索
  • LLM-as-judge · 用另一个AI模型阅读对话记录并判断处理是否安全的评分方式
  • 双方控制设定 (dual-control) · 客服AI和来电者都可以通过工具执行操作的模拟设计,而不只是客服AI单方面行动

论文原文摘要（英文）

Conversational agents now act for end users through tools while holding access to customer databases and internal policy documents that a caller can reach through dialogue alone. Banking is the clearest case: the same agent that answers a question can also change contact details, reset a PIN, or move money, so ordinary customer service is inseparable from authorization, fraud detection, and policy compliance. Existing financial-fraud benchmarks classify static transactions or messages, and general agent-safety benchmarks target prompt injection or generic harmful use; none test whether a policy-grounded banking agent safely acts when a caller manipulates identity, authorization, and trust over a conversation. We introduce FraudBench, an executable benchmark built on the $\tau^2$-bench dual-control framework and the $\tau$-Knowledge banking environment. Both the agent and the simulated caller act through tools over shared, mutable account state, and the agent may grant the caller access to selected tools; the environment exposes a 698-document internal policy corpus that the agent must retrieve from. FraudBench contains 150 authored adversarial scenarios; a frozen public set of 107 (90 across ten fraud mechanisms plus 17 chained adaptive attacks) is used for all reported runs, with 43 further chained attacks held out. Safety is history-dependent: single-control tasks satisfy every precondition but one, and adaptive attacks make a later, locally valid request unsafe because of an earlier probe, admission, or failed attempt. Each scenario is annotated with observable evidence, prohibited actions, safe dispositions, and intervention points. A preliminary single-trial evaluation of four agents on the 107 graded tasks yields attack-security between 49\% and 65\%, with money-mule and first-party fraud the most common cross-model weaknesses.

作者 · Dheeraj Mohandas Pai, Lu Xian

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Dheeraj Mohandas Pai et al., arXiv:2608.18136, CC BY 4.0