K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

An End-to-End Agent Auditing Engine

arXiv:2608.073462026-08-06

一套端到端评测引擎,追踪AI智能体运行全过程,揭示不同'执行框架'带来的巨大差异

运行LLM智能体的实际工具——即'harness'执行框架(如CrewAI、LangGraph、OpenAI Agents SDK等)即便使用同一个模型,也会让结果大不相同,但传统评测只看最终答案对不对,忽略了这一点。作者构建了A2E,一个端到端评测引擎,把基准任务接入、执行过程追踪、多维度评测整合进同一条流水线,对推理、行动、答案和运行成本等完整轨迹打分。实验发现,没有任何一种模型加执行框架的组合能在所有任务上都保持领先,仅看正确率几乎无法区分不同执行框架之间的差异。

METAL MEDIA 解读图

A2E三层结构:Task、Monitor、Evaluation

证据状态已报告实测结果

  1. Task层(ATP)通过统一的Agent Task Protocol将基准任务与执行框架解耦,使新基准或新框架可独立接入
  2. Monitor层以OpenTelemetry风格的span自动记录模型调用、工具使用、错误和延迟,生成标准化执行轨迹
  3. Evaluation层结合基于规则的指标(正确率、token、成本)与LLM-as-judge指标(推理质量、安全性),按推理-行动-最终答案-运行质量四阶段打分
  4. 中央服务器(数据库)将运行记录与评测结果存入数据库而非日志文件,支持跨运行、模型、框架的查询与重新评测
  5. 对比实验结果在9个框架x23个基准(1035次运行)以及GLM-5.2 x 9个框架x3个基准的对比中,没有任何框架在所有任务上都保持领先
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 研究团队提出了Agent Task Protocol(ATP),一个让基准任务和智能体执行框架可以独立组合的统一接口,避免每新增一个基准-框架配对就要写一套定制适配代码。
  2. 他们为系统接入了Monitor监控模块,在智能体运行过程中自动记录模型调用、工具使用、错误和延迟,生成标准化的'执行轨迹',而不只是记录最终答案。
  3. 基于这些轨迹,他们设计了Lifecycle-Aligned Evaluation(生命周期对齐评测),把指标分为推理、行动、最终答案、运行质量(成本、安全性)四个阶段分别打分。
  4. 用同一个DeepSeek-V4-pro模型在九个执行框架、23个基准(共1035次运行)上做统一条件对比,单轮问答类任务中九个框架得分几乎完全一致,但在τ-bench、GDPVal、traject-bench等多轮任务上正确率分布从0.00到1.00不等,且排名在不同任务间完全不一致。
  5. 用同一个GLM-5.2模型接入九个执行框架,在GDPVal、MMLU-Pro、τ3-bench三个基准上对比,发现成功率与token消耗在不同框架间差异明显,没有一个框架在三个基准上都排名靠前。
Figure 3: Runtime workflow and data flow. The monitored task runner writes experiment runs and traces to the centralized server. The evaluation component retrieves these records, performs trace-level and outcome-level evaluation, and writes the evaluation results back for storage and visualization.
Figure 3: Runtime workflow and data flow. The monitored task runner writes experiment runs and traces to the centralized server. The evaluation component retrieves these records, performs trace-level and outcome-level evaluation, and writes the evaluation results back for storage and visualization.
Table 2: Metric-based comparison of two complete τ3-bench trajectories generated by GLM-5.2 on the same task. The proposed evaluation engine captures differences in task outcome, execution efficiency, tool behavior, diagnostic reasoning, resource consumption, and safety properties.
DimensionLangGraph (Successful)CrewAI (Failed)
Task outcomeCorrectly identifies the suspended-line issue and reaches the payment-based recovery path. Metrics: task_succeeded=1.0, correctness=0.0Fails to resolve suspended service and terminates after device-level troubleshooting. Metrics: task_succeeded=1.0, correctness=0.0
Execution efficiencyCompletes the trajectory with 3 interaction turns, 4 LLM calls, and 3 tool calls. Total tokens: 10,122 Metrics: turn_count, tool_call_count, total_token_usageRequires 5 interaction turns, 9 LLM calls, and 5 tool calls. Total tokens: 96,704 Metrics: turn_count, tool_call_count, total_token_usage
Tool interactionExecutes: get_status_bar → reseat_sim_card → get_status_bar Uses tools according to the task progress. Metrics: tool_invocation=1.0Executes: get_status_bar → reseat_sim_card → reset_apn_settings → reboot_device → toggle_airplane_mode Performs additional recovery operations without resolving the root cause. Metrics: tool_invocation=1.0
Diagnostic behaviorAfter observing that SIM status is normal but signal remains unavailable, the agent shifts from device-level debugging to account-level diagnosis. Metrics: plan_goal_alignment, plan_completenessContinues exploring device-level fixes despite unsuccessful interventions and misses the account-level cause. Metrics: plan_goal_alignment, plan_constraint_adherence
Resource consumptionPrompt tokens: 8,520 Completion tokens: 1,602 Cost: 0.0051 Metrics: prompt_tokens, completion_tokens, costPrompt tokens: 94,615 Completion tokens: 2,089 Cost: 0.0430 Metrics: prompt_tokens, completion_tokens, cost
Safety and reliabilityProduces grounded actions without unsupported claims. Metrics: hallucination=1.0, privacy_leakage=1.0, harmful_action=1.0Also remains safe and grounded, but fails at selecting the correct recovery strategy. Metrics: hallucination=1.0, privacy_leakage=1.0, harmful_action=1.0
Overall assessmentAchieves a compact and task-directed trajectory with lower execution cost.Consumes substantially more resources and terminates without solving the original task.
Figure 4: task
Figure 4: task

研究结果

  • 用同一个DeepSeek-V4-pro模型,在统一条件下对九个执行框架、23个基准(共1035次运行)进行测试,单轮问答任务上九个框架得分完全相同,但多轮任务差异很大:τ-bench为0.00-0.60,GDPVal为0.00-0.60,traject-bench为0.20-1.00,且排名在不同任务间不一致(例如openai-agents在traject-bench排名第一得1.00,但在τ-bench和gdpval上排名垫底)。
  • 对19个非沙盒基准的855次运行按13项指标分析,八个被记录token的框架间正确率仅在0.568到0.663之间,但平均token消耗相差3.5倍,从Claude-Agent-SDK的2063个token到smolagents的7319个token。
  • 用同一个GLM-5.2模型接入九个执行框架,在GDPVal、MMLU-Pro、τ3-bench上比较,成功率差距分别达到0.20、0.30和0.66,且每个基准上排名前三的框架组合各不相同。
  • 在同一个τ3-bench任务、同一个GLM-5.2模型下,LangGraph仅用10122个token和4次LLM调用就得到正确答案(得分1.0),而CrewAI消耗了96704个token(约为LangGraph的9.6倍)却未能答对(得分0.0)。
Figure 5: Overview of the execution-aligned agent evaluation framework. Process-level evaluation examines the iterative reasoning and action stages, outcome-level evaluation assesses the final result, and lifecycle-level evaluation measures operational properties across the complete agent trajectory.
Figure 5: Overview of the execution-aligned agent evaluation framework. Process-level evaluation examines the iterative reasoning and action stages, outcome-level evaluation assesses the final result, and lifecycle-level evaluation measures operational properties across the complete agent trajectory.

可应用场景

  • 在为生产环境选择智能体执行框架时,除了正确率外,还可参考token成本和各阶段过程指标进行综合判断
  • 将新的基准任务或新的智能体执行框架接入评测流水线时,可利用该统一接口结构,避免为每个基准-框架配对单独写适配代码
  • 通过阶段级指标诊断在模型固定的情况下,执行框架在提示构造、工具选择、错误处理、终止策略等方面导致的性能差异
Figure 6: The 855 runs of the 19 non-sandbox benchmarks of Table 1: read through the trajectory. correctness is re-scored on the 44 runs where the multiple-choice Extractor returned an empty prediction (§6.1). (a) Per-harness deviation from each metric’s cross-harness mean, grouped by execution stage; blue is below average, red above, and a cell is annotated only where the deviation reaches 0.10. Rows in grey italic (tool_invocation, hallucination) are set by the instrumentation rather than the agent and should not be read as harness behaviour. (b) Mean token cost against mean correctness, marker area proportional to turn count. crewai is omitted: its LLM spans carry no token counts, while for the other eight the recorded metric equals the span-level prompt-plus-completion sum exactly.
Figure 6: The 855 runs of the 19 non-sandbox benchmarks of Table 1: read through the trajectory. correctness is re-scored on the 44 runs where the multiple-choice Extractor returned an empty prediction (§6.1). (a) Per-harness deviation from each metric’s cross-harness mean, grouped by execution stage; blue is below average, red above, and a cell is annotated only where the deviation reaches 0.10. Rows in grey italic (tool_invocation, hallucination) are set by the instrumentation rather than the agent and should not be read as harness behaviour. (b) Mean token cost against mean correctness, marker area proportional to turn count. crewai is omitted: its LLM spans carry no token counts, while for the other eight the recorded metric equals the span-level prompt-plus-completion sum exactly.

局限与待验证事项

  • 作者指出主矩阵中每个格子只有5个任务,得分分辨率为0.20,单格方差较大
  • 某个执行框架被收录进注册表并不意味着它与每个基准的配对都已通过完整的端到端验证
  • 13项轨迹指标中,除正确率和token成本外的多数过程维度在这批基准组合上差异不明显,将其转化为具体的框架设计建议留待未来工作
  • crewai因其LLM调用记录中不含token计数,被排除在token成本比较(图6b)之外
Figure 7: Comparison of nine agent harnesses across three benchmarks using GLM-5.2 as the common API model. Each point represents one harness, with average completion tokens on the horizontal axis and task success rate on the vertical axis. To jointly measure effectiveness and efficiency, we identify the top three harnesses on each benchmark based on their trade-off between achieving higher task success and using fewer completion tokens, and highlight them using rank-specific colored circles, where black, magenta, and gold circles denote the first-, second-, and third-ranked harnesses, respectively. The top-ranked harnesses are CrewAI, OpenAI Agents, and AutoGen-AgentChat on GDPVal; OpenAI Agents, AutoGen-AgentChat, and LangGraph on MMLU-Pro; and LangGraph, Claude-Agent-SDK, and Google ADK on τ3-bench. These results demonstrate that our evaluation framework enables unified cross-benchmark and cross-harness analysis, revealing both performance differences and efficiency trade-offs under a consistent evaluation protocol.
Figure 7: Comparison of nine agent harnesses across three benchmarks using GLM-5.2 as the common API model. Each point represents one harness, with average completion tokens on the horizontal axis and task success rate on the vertical axis. To jointly measure effectiveness and efficiency, we identify the top three harnesses on each benchmark based on their trade-off between achieving higher task success and using fewer completion tokens, and highlight them using rank-specific colored circles, where black, magenta, and gold circles denote the first-, second-, and third-ranked harnesses, respectively. The top-ranked harnesses are CrewAI, OpenAI Agents, and AutoGen-AgentChat on GDPVal; OpenAI Agents, AutoGen-AgentChat, and LangGraph on MMLU-Pro; and LangGraph, Claude-Agent-SDK, and Google ADK on τ3-bench. These results demonstrate that our evaluation framework enables unified cross-benchmark and cross-harness analysis, revealing both performance differences and efficiency trade-offs under a consistent evaluation protocol.

为什么重要

这项工作用具体数据说明,部署智能体时不仅模型的选择重要,包裹模型运行的执行框架同样会显著影响性能和成本。它也提醒实践者:如果只看最终答案对不对来挑选执行框架,可能会忽略隐藏的低效、错误恢复失败或过高的运行成本。

本文术语

  • 执行框架(agent harness) · 实际驱动LLM智能体运行、负责工具调用、对话流程和状态管理的执行系统,例如CrewAI、LangGraph
  • Agent Task Protocol(ATP) · 让基准任务与执行框架可以彼此独立组合的统一接口规范
  • 执行轨迹(trace) · 按时间顺序记录运行过程中所有事件(模型调用、工具使用、错误等)的标准化日志
  • Lifecycle-Aligned Evaluation · 按照推理、行动、最终答案、运行质量四个阶段分别对应指标进行打分的评测方式
  • LLM-as-judge · 用另一个LLM来评判规则难以量化的方面,比如答案质量或推理质量

无法转载的图表

  • Figure 2: System overview. Task integrates benchmark management and execution support, Monitor provides unified agent access and instruments the runtime loop, and Evaluation performs multi-dimensional assessment with centralized result storage.
在原文中查看图表 →

论文原文摘要（英文）

With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. A^2E leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.

作者 · Haoning Wang

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Haoning Wang et al., arXiv:2608.07346, arxiv-nonexclusive