DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents

arXiv:2608.185242026-08-20

训练会调用工具的AI智能体时,只精准修正它出错的那一步,效果比让它整段照抄答案还好

当一个任务包含多个可以按任意顺序完成的子目标时,正确的解法会形成一个分叉又汇合的菱形结构,但传统训练方法强迫模型死记硬背单一路径,连学生模型自己摸索出的有效替代路径也一并被惩罚。DART-SD的做法是把老师模型成功和失败的尝试都整理成一张图,找出学生失败的尝试究竟是从哪一步开始偏离了还能成功的区域(称为CTB),然后只针对这一步之后的恢复过程进行训练。在Qwen3-4B和Qwen3-8B两种规模上测试,该方法在五个工具使用基准上全面超过了现有的监督学习和强化学习方法,部分指标上甚至超过了教师模型本身。

METAL MEDIA 解读图

训练会调用工具的AI智能体时,只精准修正它出错的那一步,效果比让它整段照抄答案还好

01问题所在:包含多个可任意顺序完成的子目标的任务,其正确解法会构成分叉再汇合的菱形结构,但传统的监督微调和GRPO式强化学习都把它压平成单一线性路径,错误地惩罚了学生模型自己找到的有效替代路径
02方法:将教师模型的成功和失败尝试构建成一张交互状态转移图(ISTG),把学生失败的尝试映射到这张图上,找出它最早偏离仍可能成功区域的那个点(称为CTB),只对这个点之后生成的恢复内容计算训练损失,并反复多轮迭代进行
03验证:在FTRL数据集上训练,在FTRL、BFCL、ToolHop、τ-bench、RoTBench五个基准上评测,在Qwen3-4B和Qwen3-8B两种规模的模型上,该方法整体优于现有蒸馏方法(SCoRe-SFT、OPSD)和强化学习方法(FTRL-GRPO、ToolRL、MatchTIR)
04效率提升:随着训练轮次增加,成功轨迹的平均工具调用次数从4.23次降到3.55次,同时任务成功率持续提升,最终比人工构建的参考答案(平均4.02次)还要简洁;学生模型首次出错的位置(CTB)也随训练逐渐后移,说明其能可靠完成的步骤越来越长
05在指令遵循、数学推理、常识问答等通用能力测试中,平均得分从43.92提升到49.89,说明训练工具使用能力并未损害模型原有的其他能力

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

问题所在:包含多个可任意顺序完成的子目标的任务,其正确解法会构成分叉再汇合的菱形结构,但传统的监督微调和GRPO式强化学习都把它压平成单一线性路径,错误地惩罚了学生模型自己找到的有效替代路径
方法:将教师模型的成功和失败尝试构建成一张交互状态转移图(ISTG),把学生失败的尝试映射到这张图上,找出它最早偏离仍可能成功区域的那个点(称为CTB),只对这个点之后生成的恢复内容计算训练损失,并反复多轮迭代进行
验证:在FTRL数据集上训练,在FTRL、BFCL、ToolHop、τ-bench、RoTBench五个基准上评测,在Qwen3-4B和Qwen3-8B两种规模的模型上,该方法整体优于现有蒸馏方法(SCoRe-SFT、OPSD)和强化学习方法(FTRL-GRPO、ToolRL、MatchTIR)
效率提升:随着训练轮次增加,成功轨迹的平均工具调用次数从4.23次降到3.55次,同时任务成功率持续提升,最终比人工构建的参考答案(平均4.02次)还要简洁;学生模型首次出错的位置(CTB)也随训练逐渐后移,说明其能可靠完成的步骤越来越长
在指令遵循、数学推理、常识问答等通用能力测试中,平均得分从43.92提升到49.89,说明训练工具使用能力并未损害模型原有的其他能力

Table 1: Performance comparison of different training methods on five tool-use benchmarks using Qwen3-4B and Qwen3-8B backbones. Training-based baselines are organized into distillation-based ♠ and reinforcement learning ♢ paradigms. All trainable methods are trained on FTRL and evaluated on both the in-domain FTRL test set and four out-of-domain benchmarks. The best and second-best results within each backbone are highlighted.
Methods	FTRL	BFCL	ToolHop	τ-bench	RoTBench	Avg.
Solve-P	Solve-R	Solve-F1	Multi-Turn	AC	Pass^1	TS	PI	CF
Qwen3-4B
Base	21.00	26.54	21.81	10.14	20.20	15.15	69.52	26.31	16.07	25.19
♠ SFT	34.26	49.02	37.96	14.57	40.50	21.82	71.67	44.40	24.40	37.62
♠ SCoRe-SFT	23.79	28.20	24.61	12.75	25.83	11.52	60.36	31.19	18.33	26.29
♠ OPSD	22.37	29.21	23.50	11.00	20.20	23.03	68.93	35.24	21.67	28.35
♢ FTRL-GRPO	36.83	41.71	37.84	13.50	29.25	20.61	70.36	32.62	20.24	33.66
♢ ToolRL	26.89	33.23	28.47	9.88	20.60	18.18	69.64	38.21	24.05	29.91
♢ MatchTIR (OT)	23.55	29.27	24.95	10.38	26.23	16.36	72.02	32.74	20.36	28.43
♢ MatchTIR (KM)	25.54	31.10	26.50	10.00	26.63	21.82	72.02	33.81	21.43	29.87
♠ DART-SD (Ours)	36.70	48.16	39.77	23.88	42.11	23.03	72.02	42.38	24.52	39.17
Qwen3-8B
Base	21.18	30.71	23.48	18.38	28.54	10.13	75.52	36.29	22.19	29.60
♠ SFT	38.08	50.95	41.89	19.25	43.52	26.06	75.95	48.81	30.24	41.64
♠ SCoRe-SFT	31.28	34.48	31.58	19.25	30.25	18.18	70.36	40.48	24.52	33.38
♠ OPSD	24.37	34.88	26.68	20.50	41.11	21.21	75.24	43.21	27.38	34.95
♢ FTRL-GRPO	37.66	45.49	40.22	35.25	34.57	23.03	77.02	42.74	27.02	40.33
♢ ToolRL	32.49	41.07	35.00	20.50	44.72	25.45	75.71	51.43	33.10	39.94
♢ MatchTIR (OT)	29.45	34.89	30.65	21.38	38.79	24.24	75.00	40.36	25.36	35.57
♢ MatchTIR (KM)	33.07	40.42	35.37	23.25	41.71	26.06	75.71	42.26	26.90	38.31
♠ DART-SD (Ours)	42.00	54.13	45.66	27.63	45.03	27.12	75.83	57.38	35.48	45.58

Table 2: Average tool-call length of successful trajectories across progressive SFT iterations on the FTRL test set. DART-SD progressively improves Solve-F1 while shortening its tool traces, eventually producing more efficient traces than the golden solutions given during data construction.
Task	DART-SD	Golden
Iter1	Iter2	Iter3	Iter4	Iter5
Single	1.23	1.15	1.10	1.09	1.07	1.00
Multi	3.83	4.63	4.54	4.49	4.41	4.71
Para-Single	3.29	2.65	2.27	2.26	2.22	2.11
Para-Multi	7.25	6.48	5.97	5.82	5.62	6.97
Overall	4.23	3.99	3.71	3.65	3.55	4.02
Solve-F1	40.37	42.95	43.78	44.67	45.66	–

Table 3: Average CTB positions of failed training trajectories across progressive SFT. Larger values indicate that the first departure from empirically recoverable behavior occurs later, meaning that the model correctly executes a longer trajectory prefix before localized recovery is required.
Task	CTB Position
Iter1	Iter2	Iter3	Iter4	Iter5
Single	0.034	0.186	0.137	0.249	0.077
Multi	0.500	1.624	1.729	1.876	1.953
Para-Single	0.095	0.219	0.299	0.328	0.313
Para-Multi	0.395	1.524	1.692	1.791	1.816
Overall	0.348	1.185	1.310	1.421	1.452
Δ vs. Iter1	–	+0.837	+0.962	+1.073	+1.104

Table 4: Performance comparison under the thinking setting. Results are reported using FTRL Solve-F1, the average BFCL Multi-Turn score, and ToolHop AC.
Method	FTRL	BFCL	ToolHop
Qwen3-8B	29.74	40.00	42.21
FTRL-GRPO	32.85	41.50	36.72
ToolRL	26.72	34.25	32.93
MatchTIR (KM)	37.33	47.13	46.16
DART-SD	41.03	49.75	46.43

Table 5: General capability evaluation on representative benchmarks. The best results are highlighted.
Method	IFEval	AIME24	AIME25	MMLU	Avg.
Qwen3-8B	34.75	46.67	23.33	70.94	43.92
SFT	35.30	43.33	26.67	71.43	44.18
DART-SD	45.29	50.00	30.00	74.27	49.89

Table 6: Component ablation of DART-SD on the FTRL test set using Solve-P, Solve-R, and Solve-F1.
Method	Solve-P	Solve-R	Solve-F1
Qwen3-8B	21.18	30.71	23.48
+SD	36.23	44.46	38.10
+CTB	36.62	46.32	39.51
+Progressive SFT	41.32	49.65	43.93
+ISTG (Ours)	42.00	54.13	45.66

为什么重要

这项工作说明,训练需要反复调用工具解决问题的AI智能体时,强迫模型死记硬背一条固定正确路径既低效,又可能破坏模型本身已有的有效解题方式。精准定位并只修正出错的那一步,为打造更小、更聪明、更高效的工具调用智能体提供了一条实用路径。

本文术语

监督微调(SFT) · 让模型直接模仿给定示例答案的额外训练方式
GRPO · 一种利用奖励信号调整模型行为的强化学习方法
交互状态转移图(ISTG) · 用节点表示已获取信息、记录工具调用过程的图结构
CTB(关键拓扑分歧点) · 学生模型尝试中首次偏离仍可能成功路径的那一个位置
自我蒸馏 · 利用模型自己生成的结果重新训练自身的方法

无法转载的图表

Figure 1: Comparison of training paradigms. (a) SFT Teacher Boosting applies an indiscriminate global loss, which overwrites valid exploration. (b) Standard-RL (GRPO) misassigns credit through a uniformly distributed reward spread. (c) DART-SD dynamically identifies the Critical Topological Breakpoints (CTB) and applies localized correction while preserving valid exploration.
Figure 2: Overview of DART-SD. (1) DART-SD constructs an ISTG from teacher rollouts, where main and auxiliary nodes model information acquisition and useless exploration. (2–3) Failed student rollouts are collected and replayed in the same interaction-state space, then projected onto the budget-filtered success-reachable region ℛx+, where the first projectable-to-non-projectable transition defines the CTB. (4–5) Conditioned on the retained student prefix and privileged teacher references, DART-SD generates a recovery continuation after the CTB and applies localized supervision only to the generated assistant tokens. (6) This CTB-guided SFT loop is progressively repeated across self-distillation rounds.
Figure 3: Performance comparison of Qwen3-8B, DART-SD, and the teacher across five tool-use benchmarks. DART-SD improves upon Qwen3-8B on all benchmarks and surpasses the teacher on FTRL, ToolHop, and τ-bench.

在原文中查看图表 →

论文原文摘要（英文）

Equipping Large Language Models (LLMs) with multi-turn tool-calling capabilities is essential for building autonomous agents. However, progress is fundamentally limited by the reliance on full-length trajectory imitation. For tasks involving multiple order-independent sub-goals, the optimal solution space forms a vast combinatorial diamond lattice. Forcing this rich topology into monolithic trajectories causes a severe topological collapse, indiscriminately penalizing valid alternative explorations and severely degrading policy diversity. To address this, we propose DART-SD (Diamond-topology Aware Retrieval and Tuning for Self-Distillation), a novel framework that shifts the paradigm from global forcing to topology-guided localized correction. DART-SD first models the execution process as a converging Interaction-State Transition Graph (ISTG), faithfully capturing the inherent diamond topology of successful and failed exploratory paths. During autonomous rollouts, the framework identifies the Critical Topological Breakpoint (CTB) and retrieves success-supported recovery references. Finally, we introduce a progressive self-distillation paradigm through CTB-guided localized supervision, ensuring that the training loss is calculated exclusively on the generated recovery steps while strictly protecting the valid reasoning prefix from destructive gradient updates. Experiments on complex multi-turn tool-calling benchmarks demonstrate that DART-SD significantly outperforms traditional full-trajectory baselines.

作者 · Hangrui Xu, Jiarui Wang, Yang Yang, Chuanbo Zhu, Fangda Chen, Ziqi Wu, Jingming Cai, Yan Song

在 arXiv 阅读

DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents

训练会调用工具的AI智能体时,只精准修正它出错的那一步,效果比让它整段照抄答案还好

他们做了什么

为什么重要

本文术语

无法转载的图表

最新论文

METAL MEDIA 最新报道