Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

arXiv:2608.194082026-08-21

AI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤

在线策略蒸馏(On-Policy Distillation, OPD)让学生模型根据自己生成的答案对照老师模型打分学习,但这种打分方式有个盲点:即使学生的推理步骤正在真正接近正确答案,只要写法和老师不一样就会被扣分。研究团队提出R2-OPD,专门独立衡量每一步推理是否真的在接近答案,当老师打分和真实进展方向相矛盾时,就把这一段的扣分信号过滤掉。在DeepSeek-R1-Distill-Qwen-1.5B上的实验中,相比标准OPD,平均准确率(avg@4)提升2.51分,四次尝试中至少答对一次的比例(pass@4)提升4.46分。

METAL MEDIA 解读图

AI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤

01研究发现OPD按token逐一对照老师模型分布打分的方式,并不总能准确反映学生是否真的在朝正确答案推进
02为独立衡量真实进展,团队从每个中间推理节点出发做Neval=8次模拟续写,估算学生从这一步能答对的概率,由此计算出真正的进展分数(process reward)
03将进展方向一致的相邻推理片段合并(sign-consistent merging)以降低噪声,再比较进展分数排序与老师相似度分数排序是否矛盾,对矛盾片段的老师监督信号进行屏蔽(masking)
04在以JustRL为老师模型的DeepSeek-R1-Distill-Qwen-1.5B实验中,R2-OPD相比标准OPD平均准确率(avg@4)提升2.51分,pass@4提升4.46分,在AIME数学竞赛题上提升尤为明显
05该方法在Qwen3-1.7B学生模型搭配e3-1.7B老师模型的组合上同样有效,说明方法能跨模型系列迁移

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

研究发现OPD按token逐一对照老师模型分布打分的方式,并不总能准确反映学生是否真的在朝正确答案推进
为独立衡量真实进展,团队从每个中间推理节点出发做Neval=8次模拟续写,估算学生从这一步能答对的概率,由此计算出真正的进展分数(process reward)
将进展方向一致的相邻推理片段合并(sign-consistent merging)以降低噪声,再比较进展分数排序与老师相似度分数排序是否矛盾,对矛盾片段的老师监督信号进行屏蔽(masking)
在以JustRL为老师模型的DeepSeek-R1-Distill-Qwen-1.5B实验中,R2-OPD相比标准OPD平均准确率(avg@4)提升2.51分,pass@4提升4.46分,在AIME数学竞赛题上提升尤为明显
该方法在Qwen3-1.7B学生模型搭配e3-1.7B老师模型的组合上同样有效,说明方法能跨模型系列迁移

Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).

Table 1: OPD performance on DeepSeek-R1-Distill-Qwen-1.5B with JustRL as the teacher model.
Method	AIME 24	AIME 25	Olympiad	Avg.
avg@4	pass@4	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4
Student	22.50	43.33	23.33	36.67	43.19	58.31	29.67	46.10
Teacher	41.67	56.67	30.83	43.44	53.28	68.43	41.92	56.18
OPD (1)	28.33	50.00	22.50	30.00	46.86	62.10	32.55	47.37
E-OPD (17)	18.33	36.67	12.50	23.33	49.91	64.56	26.91	41.52
TIP-OPD (36)	17.50	33.33	11.67	20.00	48.24	62.83	25.80	38.72
IW-OPD (34)	20.83	36.67	17.50	26.67	43.59	59.09	27.31	40.81
Uni-OPD (13)	20.00	43.33	19.17	26.67	53.16	69.97	30.78	46.66
R2-OPD (Ours)	32.50	56.67	25.83	36.67	46.86	62.19	35.06	51.83

Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment (J9) receives a smaller KL divergence under Equation (9) than segments that derive or recover the correct solution, respectively(J2 and J10).

Table 2: Transfer performance of R2-OPD on Qwen3-1.7B with e3-1.7B as the teacher model.
Dataset	Base	OPD	R2-OPD
avg@4	pass@4	avg@4	pass@4	avg@4	pass@4
AIME 24	24.17	30.00	22.50	36.67	25.00	40.00
AIME 25	18.75	20.00	27.50	33.33	25.83	36.67
Olympiad	50.10	63.87	53.82	67.09	54.31	67.91
Avg.	31.01	37.96	34.61	45.70	35.04	48.19

Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.

Table 3: Effect of sign-consistent segment merging on avg@4 accuracy. Δ denotes the improvement of the full R2-OPD over its no-merging variant.
Dataset	No Merge	R2-OPD	Δ
AIME 24	17.5	32.50	15.0
AIME 25	11.67	25.83	14.16
Olympiad	47.8	46.86	-0.96

Figure 4: Sensitivity of R2-OPD to the masking ratio q. A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.

Table 4: Optimization, process-reward estimation, and filtering settings used in all R2-OPD experiments.
Setting	Value
Optimization and sequence settings
Training epochs	1
Optimizer	AdamW
Learning rate	5×10−6
Global batch size	64
Maximum prompt length	1,024 tokens
Maximum response length	7,168 tokens
KL support size	Student top-16 (H=16)
Process-reward estimation
Rollouts per evaluated boundary	Neval=8
Sampling temperature	0.7
Top-k	50
Top-p	1.0
Maximum rollout length	300 tokens
Segment filtering
Minimum sentences between boundaries	Smin=3
Minimum merged segments	nmin=3
Segment masking ratio	q=30%

Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.

Table 5: Eligibility checks and conservative fallback actions used by R2-OPD.
Condition	Action
The string-level pre-check does not find gi in yi, or segmentation yields fewer than two segments.	Skip process-reward rollouts and retain all token-level OPD supervision.
The response has fewer than max⁡(3,nmin) merged segments.	Do not apply segment masking.
No strict adjacent pair is available after process-reward ranking.	Do not apply segment masking.
All candidate segments have zero inconsistency score.	Do not apply segment masking.
At least one candidate has a positive inconsistency score.	Mask up to the response-specific budget; retain every other token.

Table 6: Complete lexicon of discourse markers used to propose reasoning-segment boundaries.
Category	Matched markers
Reconsideration	wait, hold on, let me reconsider, hmm
Correction	actually
Verification	let me check
Alternative reasoning	alternatively

为什么重要

这项研究揭示了传统师生蒸馏训练的一个结构性缺陷:只因写法和老师不同就压制学生本可能正确、有创意的推理路径。对于希望更便宜、更有效地训练数学或编程等推理型AI模型的团队,这提供了可直接借鉴的训练改进思路。

本文术语

在线策略蒸馏(On-Policy Distillation, OPD) · 让学生模型用自己生成的答案接受老师模型打分来学习的一种知识蒸馏方法
反向KL散度(reverse KL divergence) · 衡量学生模型概率分布与老师模型差异程度的指标,常被用作训练信号
进展分数(process reward) · 衡量某一步推理相比前一步,答对最终问题概率提升了多少的分数
方向一致合并(sign-consistent merging) · 把进展方向相同的相邻推理片段合并起来,以降低测量噪声
avg@4 / pass@4 · avg@4是四次尝试的平均准确率,pass@4是四次尝试中至少答对一次的比例

论文原文摘要（英文）

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

作者 · Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Chen Yang et al., arXiv:2608.19408, arxiv-nonexclusive

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

AI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤

他们做了什么

为什么重要

本文术语

最新论文

METAL MEDIA 最新报道