Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

arXiv:2607.266272026-07-28

为大模型解码提速的“有损验证”技巧,实际上会悄悄扭曲输出分布并损害生成质量

推测解码(Speculative Decoding)让一个小的草稿模型先提出候选词,再由大的目标模型并行验证,从而加速大语言模型推理;最近出现的“有损验证”方法通过放宽验证标准进一步提速,但本文发现这些方法实际上只归为两大机制,一旦与正确的基线对比,都可能比之前报告的更严重地损害生成质量。代码已在GitHub的Fast-HSD仓库中开源。

METAL MEDIA 解读图

有损验证为何会悄悄扭曲输出质量

证据状态已报告实测结果

1. 基线:推测解码小草稿模型q提出候选词,大目标模型p并行验证;无损验证保证最终输出分布与p完全一致。
2. 两大有损验证家族作者将现有方法归为基于截断的验证(落入允许集合即接受)和协作式验证(将p与q混合为组合分布)两类。
3. 截断验证的陷阱只要落在允许集合内就一律接受,导致最终输出分布更像草稿模型q而非目标模型p,该扭曲随任务难度增加、以及在EAGLE-3树验证下进一步加剧。
4. 协作式验证的关键:限制超冲只对草稿概率远超目标概率的“超冲”词元设置上限,比在整个词表上均匀混合p和q更能保持生成质量。
5. 结论:需要用匹配基线比较有损验证方法看似带来的加速,应当与把同一截断/协作规则直接用于目标模型的匹配基线比较,而非与默认解码比较。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

推测解码通过让轻量级草稿模型提出候选词、再由更大的目标模型并行验证来加速大语言模型推理;近期出现的“有损验证”方法进一步放宽严格的分布匹配要求以换取更快速度。
作者证明,看似不同的多种有损验证方法实际上只可归为两类:基于截断的验证(只要候选词落在某种截断采样规则定义的允许集合内就直接接受)和协作式验证(将草稿分布与目标分布混合)。
在MATH、MBPP+、INCLUDE、BFCL四个基准上,使用Qwen2.5-72B/0.5B组合以及配合EAGLE-3的LLaMA-3.1 8B进行实验,基于截断的方法(SpecCascade、typical acceptance)始终不如把同一截断采样规则直接用在目标模型上的匹配基线,且这一差距随任务难度增大、以及在EAGLE-3多候选树验证下进一步扩大。
针对协作式验证的消融实验发现,只对草稿模型概率“超冲”(overshoot,即草稿给出的概率远高于目标模型)的部分设置上限,比像CoS那样将两个分布均匀混合更能保持生成质量,而均匀混合会导致质量严重下降。

Figure 1: Accuracy gap between the lossless baseline and truncation-based verification widens sharply with task difficulty, increasing from +0.38 pp on GSM8K to +6.67 pp on AIME. Here, the True baseline means the target model with min-p sampling, while the Wrong baseline indicates the target model w/o min-p sampling.

Table 1: Ablation of the two mechanisms in lenience-based relaxation on MBPP+.
Variant	𝝀	BE	Pass@1 (%)
Adaptive interpolation	0.2	9.15	50.26
0.4	7.93	56.61
0.6	6.80	60.85
0.8	5.95	66.14
Overshoot ceiling	0.2	5.59	75.93
0.4	5.57	75.13
0.6	5.54	75.66
0.8	5.54	75.13

Figure 2: Comparison of distributions induced by collaborative and truncation-based methods. Tokens are sorted by decreasing target probability. Blue and pink bars denote the target distribution p and draft distribution q, respectively; orange denotes their overlap. The purple line shows the output distribution. Gray dashed lines indicate the ceiling.

Table 2: Evaluation results using Qwen2.5-72B and Qwen2.5-0.5B pair under standard speculative decoding averaged over all hyperparameter settings. To ensure a fair comparison, each verification method is compared only with its matched truncation sampling baseline using the same allowed set. Bold denotes the better task performance within each matched pair. Gray denotes the SD baseline, which is not directly comparable.
Method	MATH	MBPP+	INCLUDE	BFCL
BE	DS	Acc (%)	BE	DS	Pass@1 (%)	BE	DS	Acc (%)	BE	DS	Acc (%)
Standard SD (14)	7.98	4.67	76.47	5.47	4.06	75.84	3.40	4.63	68.18	8.73	6.86	88.17
Min-p sampling (20)	8.01	4.64	76.51	5.53	3.89	75.87	3.44	4.62	67.79	8.85	6.96	87.97
SpecCascade (19)	8.23	4.62	75.63	5.53	3.90	75.88	3.58	4.62	66.82	8.86	6.52	88.30
η-sampling (12)	7.98	4.61	76.14	5.50	3.89	75.85	3.36	4.61	68.18	8.78	6.82	88.17
Typical acceptance (4)	8.39	4.66	75.55	5.62	3.89	75.80	3.58	4.62	66.79	8.87	6.83	88.93

Figure 3: Efficiency and task performance tradeoff for collaborative verification across four benchmarks. Each marker represents one hyperparameter setting, and lines connect settings in sweep order. The star and dashed reference lines denote the lossless speculative-decoding baseline. See Appendix C for complete results.

Table 3: Evaluation results using LLaMA-3.1 8B and its official released pair under EAGLE-3 speculative decoding, averaged across the parameter grid for each method (T=0.7, block size =7). Bold denotes the better task performance within each matched pair. Gray denotes the EAGLE-3 baseline, which is not directly comparable.
Method	MATH	MBPP+	INCLUDE	BFCL
BE	DS	Acc (%)	BE	DS	Pass@1 (%)	BE	DS	Acc (%)	BE	DS	Acc (%)
EAGLE-3 (16)	3.76	140.21	73.20	4.70	167.83	59.30	0.67	45.72	35.50	2.57	85.27	86.00
Min-p sampling (20)	3.98	143.22	76.88	4.90	168.57	61.80	0.56	45.80	37.00	2.58	84.26	86.60
SpecCascade (19)	4.22	152.74	76.16	5.00	175.91	60.74	0.78	48.13	33.27	2.59	85.20	85.40
η-sampling (12)	3.93	140.05	76.12	4.87	165.61	62.17	0.55	45.33	36.73	2.58	79.77	86.30
Typical acceptance (4)	4.61	162.73	69.84	5.20	180.10	56.88	1.08	55.54	27.91	2.64	85.61	81.40

Figure 4: Net change in ΔBE under different draft–target alignment ratios for Min-p and η-sampling. Each triplet reports the ratios of matching, partially overlapping, and unrelated candidate tokens, respectively.

Table 4: Verification method results across the four benchmarks (MATH, MBPP+, INCLUDE, BFCL). For each method we sweep its characteristic hyperparameter (Param) and report block efficiency (BE), decoding speed (DS), and accuracy / Pass@1; all entries are mean ± std over three seeds.
Method	Param	MATH	MBPP+	INCLUDE	BFCL
BE	DS	Acc (%)	BE	DS	Pass@1 (%)	BE	DS	Acc (%)	BE	DS	Acc (%)
SD Baseline 14	–	7.98±0.02	4.67±0.02	76.47±1.53	5.47±0.06	4.06±0.03	75.84±0.40	3.40±0.03	4.63±0.01	68.18±0.00	8.73±0.01	6.86±0.00	88.17±0.58
Min-p Smpl. + tokenwise SpD	0.1	7.94±0.02	4.65±0.02	76.27±1.53	5.42±0.08	3.89±0.14	75.57±0.40	3.34±0.04	4.63±0.01	68.79±0.69	8.73±0.01	6.97±0.18	88.17±0.58
0.3	7.99±0.05	4.64±0.01	76.67±0.83	5.56±0.03	3.90±0.14	76.19±0.46	3.42±0.04	4.62±0.01	68.18±2.08	8.85±0.01	6.94±0.21	88.33±1.04
0.5	8.03±0.02	4.64±0.01	76.07±1.17	5.45±0.07	3.89±0.14	75.57±0.31	3.44±0.02	4.61±0.02	68.48±1.46	8.88±0.02	6.97±0.19	87.67±0.29
0.7	8.08±0.04	4.64±0.01	76.27±0.42	5.59±0.02	3.88±0.13	76.01±0.31	3.47±0.05	4.63±0.01	66.52±1.14	8.89±0.02	6.96±0.19	87.67±0.29
0.9	8.08±0.10	4.65±0.01	77.27±1.27	5.62±0.04	3.88±0.14	76.01±0.15	3.47±0.04	4.62±0.01	66.97±1.31	8.89±0.01	6.97±0.18	88.00±0.00
Cascade 5	0.1	8.46±0.06	4.53±0.22	76.87±0.46	5.64±0.03	3.89±0.14	76.46±0.26	3.57±0.04	4.63±0.01	64.85±1.14	8.87±0.04	6.83±0.05	89.00±0.50
0.3	8.36±0.04	4.65±0.01	75.47±0.70	5.59±0.04	3.89±0.14	75.40±0.26	3.50±0.03	4.60±0.01	66.67±3.44	8.89±0.01	6.81±0.02	88.67±0.29
0.5	8.12±0.05	4.66±0.01	74.87±0.42	5.51±0.03	3.90±0.12	74.87±0.70	3.40±0.08	4.61±0.02	68.33±3.03	8.85±0.03	6.88±0.03	88.00±0.00
0.7	8.01±0.06	4.66±0.01	75.47±0.70	5.48±0.01	3.90±0.14	76.46±0.53	3.30±0.04	4.60±0.01	68.33±1.05	8.84±0.03	6.82±0.03	87.67±0.29
0.9	7.84±0.07	4.66±0.01	75.47±1.40	5.41±0.01	3.89±0.14	76.19±0.26	3.25±0.03	4.62±0.01	65.91±0.45	8.84±0.03	5.28±0.04	88.17±0.58
η Smpl. + tokenwise SpD	0.05	7.92±0.04	4.63±0.01	76.67±1.85	5.42±0.08	3.89±0.14	75.57±0.40	3.34±0.05	4.61±0.02	67.73±1.82	8.73±0.01	6.95±0.18	88.17±0.58
0.10	7.94±0.03	4.64±0.01	76.47±1.03	5.49±0.06	3.88±0.15	76.54±0.61	3.31±0.02	4.62±0.02	68.79±0.26	8.74±0.01	6.82±0.02	88.17±0.58
0.15	7.96±0.03	4.64±0.02	75.87±2.21	5.44±0.02	3.89±0.15	76.01±0.31	3.39±0.05	4.60±0.03	67.27±0.45	8.77±0.01	6.79±0.05	88.17±0.58
0.20	8.11±0.29	4.51±0.21	76.27±0.70	5.57±0.05	3.88±0.12	76.01±0.15	3.42±0.05	4.61±0.01	68.33±1.84	8.80±0.07	6.80±0.02	88.17±0.58
0.25	8.03±0.06	4.50±0.21	75.40±1.60	5.57±0.04	3.87±0.14	75.13±0.26	3.41±0.02	4.63±0.03	68.79±1.60	8.88±0.05	6.72±0.07	88.17±0.58
Medusa 4	0.05	8.49±0.01	4.67±1.93	76.60±2.08	5.64±0.03	3.89±0.14	76.46±1.06	3.58±0.03	4.62±0.01	66.52±1.31	8.87±0.04	6.84±0.02	89.00±0.50
0.10	8.47±0.05	4.66±0.00	75.20±1.11	5.64±0.03	3.89±0.13	76.46±1.06	3.63±0.03	4.61±0.01	66.52±1.05	8.87±0.04	6.85±0.01	89.00±0.50
0.15	8.21±0.24	4.65±0.03	75.73±2.12	5.66±0.03	3.88±0.14	75.66±0.53	3.59±0.01	4.62±0.02	66.06±1.31	8.87±0.04	6.85±0.03	89.00±0.50
0.20	8.39±0.01	4.65±0.00	74.67±1.55	5.63±0.05	3.90±0.14	75.40±0.26	3.54±0.03	4.60±0.01	68.79±0.69	8.87±0.04	6.84±0.02	89.00±0.50
0.25	8.30±0.05	4.67±0.01	75.53±0.31	5.55±0.08	3.90±0.14	75.04±0.15	3.46±0.01	4.63±0.01	66.06±0.26	8.89±0.01	6.78±0.04	88.67±0.29
Lenience-based relaxation	0.2	8.49±0.02	4.68±0.05	74.47±0.58	5.65±0.03	4.06±0.02	75.13±0.26	3.59±0.04	4.52±0.04	67.42±1.39	8.87±0.01	6.90±0.02	88.67±0.29
0.4	8.37±0.08	4.66±0.12	75.60±2.65	5.61±0.02	4.06±0.02	75.49±0.67	3.55±0.06	4.53±0.09	67.88±1.31	8.85±0.04	6.87±0.01	88.00±0.50
0.6	8.26±0.05	4.66±0.14	78.00±1.73	5.52±0.07	4.06±0.03	75.57±0.40	3.48±0.02	4.50±0.18	68.64±1.82	8.83±0.02	6.84±0.02	88.17±0.29
0.8	8.22±0.07	4.65±0.23	76.47±1.15	5.52±0.04	4.06±0.02	75.75±0.40	3.38±0.03	4.49±0.11	68.33±0.26	8.81±0.05	6.87±0.02	87.67±0.29
CoS 9	0.2	8.33±0.00	4.68±0.01	71.53±2.12	6.05±0.13	4.10±0.01	71.43±1.06	3.89±0.04	4.61±0.02	64.55±1.98	8.76±0.13	6.79±0.04	70.67±3.40
0.4	8.80±0.04	4.67±0.00	60.40±2.84	7.14±0.09	4.13±0.01	61.20±3.29	4.69±0.07	4.58±0.03	53.79±2.10	8.84±0.15	6.69±0.03	58.00±4.50
0.6	9.42±0.01	4.67±0.01	54.80±0.72	7.86±0.02	4.20±0.02	59.44±0.85	5.99±0.03	4.53±0.01	48.03±1.89	9.29±0.16	6.69±0.03	43.83±3.79
0.8	10.14±0.11	4.65±0.01	46.00±2.12	9.26±0.04	4.26±0.00	54.06±0.31	8.02±0.05	4.49±0.01	37.12±2.05	10.26±0.07	6.75±0.01	43.67±1.89

Figure 5: Block efficiency of truncation-based methods across hyperparameter settings on four benchmarks. Top row: Min-p sampling and SpecCascade as pbase varies. Bottom row: η-sampling and typical acceptance as ϵ varies. Lines show the mean and shaded regions indicate one standard deviation. The dashed line denotes the lossless speculative-decoding baseline under the default Qwen2.5 sampling parameters.

Table 5: EAGLE-3 speculative decoding results across four benchmarks (LLaMA-3.1 8B, temperature =0.7, block size=7). Bold marks the best value per column across all methods; underline marks the second best.
		MATH	MBPP+	INCLUDE	BFCL
Method	Param	BE	DS	Acc (%)	BE	DS	Pass@1 (%)	BE	DS	Acc (%)	BE	DS	Acc (%)
Baseline	—	3.76	140.21	73.20	4.70	167.83	59.30	0.67	45.72	35.50	2.57	85.27	86.00
Lenience 14	0.2	4.49	161.29	71.20	5.17	181.95	61.11	0.86	50.61	30.91	2.63	87.31	83.50
0.4	4.34	157.03	76.20	5.02	177.13	59.79	0.80	48.93	32.73	2.61	86.14	85.50
0.6	4.19	152.85	76.00	4.96	175.43	59.52	0.73	47.03	34.55	2.59	85.77	86.00
0.8	4.07	149.39	75.20	4.92	174.25	59.79	0.72	46.73	33.64	2.59	85.56	84.00
SpecCascade 19	0.1	4.47	159.83	73.00	5.16	180.81	59.79	0.92	51.76	30.91	2.62	85.82	84.50
0.3	4.28	154.42	77.40	5.03	176.79	61.38	0.79	48.26	31.82	2.59	85.18	84.50
0.5	4.18	151.56	77.60	4.99	175.66	60.85	0.76	47.43	31.82	2.58	85.03	86.50
0.7	4.12	149.71	76.20	4.93	173.70	61.38	0.73	46.74	36.82	2.58	84.99	85.50
0.9	4.06	148.20	76.60	4.89	172.60	60.32	0.72	46.45	35.00	2.58	85.00	86.00
Min-p Smpl. + SD	0.1	3.89	140.72	75.20	4.84	166.79	61.90	0.54	45.34	40.00	2.58	84.45	86.00
0.3	3.96	142.85	77.60	4.89	168.29	60.32	0.55	45.65	36.82	2.58	84.13	86.00
0.5	3.98	143.42	77.40	4.91	168.98	60.85	0.56	45.91	35.91	2.58	84.09	86.50
0.7	4.00	143.88	77.40	4.92	169.23	62.43	0.56	45.96	35.91	2.59	84.21	87.50
0.9	4.05	145.24	76.80	4.93	169.58	63.49	0.57	46.13	36.36	2.59	84.42	87.00
η Sampl. + SD	0.05	3.85	137.92	75.20	4.84	164.79	59.79	0.55	45.42	35.00	2.58	83.63	87.50
0.10	3.90	139.33	77.00	4.86	165.32	61.11	0.54	45.06	41.82	2.59	83.92	86.00
0.15	3.93	140.21	77.20	4.88	165.95	61.90	0.54	45.13	35.00	2.59	83.91	86.50
0.20	3.96	140.97	76.00	4.89	166.23	64.29	0.54	45.14	34.09	2.57	83.54	85.00
0.25	3.99	141.83	75.20	4.88	165.77	63.76	0.57	45.92	37.73	2.58	63.85	86.50
Typical Sampling	0.05	4.82	168.77	66.00	5.29	182.86	55.29	1.24	59.80	29.55	2.66	86.11	78.00
0.10	4.66	164.19	68.20	5.21	180.54	56.08	1.15	57.20	29.55	2.65	85.85	79.50
0.15	4.58	161.86	72.20	5.16	178.87	57.41	1.05	54.76	25.91	2.64	85.65	83.00
0.20	4.54	160.65	69.80	5.16	178.94	58.47	0.98	52.73	27.73	2.63	85.40	83.50
0.25	4.46	158.17	73.00	5.17	179.31	57.14	1.00	53.21	26.82	2.62	85.05	83.00

Figure 6: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on four benchmarks. Each point represents a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling.

Table 6: Effect of truncation and overshoot capping on MBPP+, averaged over ℓ∈{0.2,0.4,0.6,0.8}. pbase=0.5 is used in 𝒜Θ.
Yielded rule	Avg. BE	Avg. Pass@1 (%)
P⁡(generatex)=p⁡(x)	5.42	75.66
P⁡(generatex)={q⁡(x),x∈𝒜Θ,0,otherwise,	5.51 (+1.7%)	74.87 (-1.0%)
P⁡(generatex)={Δp(x)+(1−Δ)q(x),q⁡(x)≤p⁡(x),q⁡(x),p⁡(x)<q⁡(x)≤p⁡(x)/ℓ,p⁡(x)/ℓ,q⁡(x)≥p⁡(x)/ℓ,	5.53 (+2.0%)	75.22 (-0.6%)
P⁡(generatex)={q⁡(x),q⁡(x)≤p⁡(x)/ℓ,p⁡(x)/ℓ,q⁡(x)≥p⁡(x)/ℓ,	5.58 (+3.0%)	75.33 (-0.4%)
P⁡(generatex)={q⁡(x),x∈𝒜Θ,min⁡{q⁡(x),p⁡(x)/ℓ},otherwise	5.62 (+3.7%)	75.66 (-0.0%)

Figure 7: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on three benchmarks. Each point shows a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling. Detailed results can be found in Appendix C.

Table 7: Gap of truncation-based verification relative to its matched truncation sampling + SD baseline (Δ= lossy − lossless), without EAGLE-3 (single-draft SD, Table 2) and with EAGLE-3 (Table 3). EAGLE-3 amplifies the average task-performance deficit by roughly 4× (SpecCascade) and 20× (typical acceptance).
Matched pair	Framework	MATH	MBPP+	INCLUDE	BFCL	Avg.
ΔBE	ΔAcc	ΔBE	ΔAcc	ΔBE	ΔAcc	ΔBE	ΔAcc	ΔBE	ΔAcc
SpecCascade − Min-p sampling	Single-draft SD	+0.22	−0.88	+0.00	+0.01	+0.14	−0.97	+0.01	+0.33	+0.09	−0.38
EAGLE-3	+0.24	−0.72	+0.10	−1.06	+0.22	−3.73	+0.01	−1.20	+0.14	−1.68
Typical acceptance − η-sampling	Single-draft SD	+0.41	−0.59	+0.12	−0.05	+0.22	−1.39	+0.09	+0.76	+0.21	−0.32
EAGLE-3	+0.68	−6.28	+0.33	−5.29	+0.53	−8.82	+0.06	−4.90	+0.40	−6.32

Figure 8: Two views of lenience-based relaxation. Fig.8(a) illustrates how the interpolation strength adapts to the draft–target divergence on a per-token basis. Fig.8(b) contrasts a global linear mixture with a pointwise ceiling at p/ℓ, highlighting the asymmetric treatment of overshoot.

研究结果

在标准推测解码下,基于截断的方法(SpecCascade、typical acceptance)在全部四个基准上均不如直接把同一截断采样规则用在目标模型上的匹配基线,匹配对之间的差距最大为1.4分。
在EAGLE-3的多候选树验证下,这一差距明显扩大:SpecCascade的平均差距从-0.38分扩大到-1.68分,typical acceptance从-0.32分扩大到-6.32分,在INCLUDE基准上差距最大达到-8.8分。
无损基线与基于截断验证之间的准确率差距,从较简单的GSM8K基准上的+0.38个百分点,扩大到较难的AIME基准上的+6.67个百分点。
在MBPP+上的消融实验表明,基于宽容度(lenience)的松弛方法之所以有效,关键在于对“超冲”区域设置上限(p/l),而不是在低估区域使用自适应插值;仅使用超冲上限就能在保持与无损验证相当的任务表现的同时提升效率。
一种结合规则——用min-p允许集合来控制接受、并在集合外应用超冲上限——相比无损推测解码在区块效率上提升了3.7%,同时Pass@1准确率完全保持一致。

可应用场景

部署推测解码以加速大语言模型推理的团队,可以用这一框架检验某种有损验证方法报告的加速效果,是否真的优于直接把同一截断采样规则用在目标模型上的结果。
在代码生成、数学推理、多语言理解、工具调用等对精度要求较高的任务中,该分析有助于判断哪种验证方案的质量下降风险更低。
在考虑将有损验证与EAGLE-3等基于树结构的多候选验证系统结合时,团队应意识到质量下降幅度可能远大于标准单候选推测解码。
设计新的协作式验证方案时,可以优先考虑选择性抑制超冲词元,而非对草稿和目标分布进行均匀混合。

局限与待验证事项

实验仅限于Qwen2.5和Llama-3.1模型家族,且目标模型与草稿模型的配对是固定的,截断陷阱的程度和超冲原则是否能推广到其他架构、模型规模或目标-草稿比例尚未得到验证。
评估集中在推理、代码、多语言和函数调用类基准(MATH、MBPP+、INCLUDE、BFCL)上,开放式生成和对话场景(质量更难自动衡量)不在本研究范围内。
理论分析基于论文中形式化的特定截断与协作规则,未必能直接覆盖未来可能出现的其他有损验证方案。
区块效率被用作与硬件无关的速度替代指标,但实际的墙钟加速效果取决于硬件和服务框架,真实的速度-质量权衡会因部署环境不同而变化。

为什么重要

对于正在使用推测解码来加速大语言模型推理的团队,这项研究提醒:论文中报告的加速效果可能把截断采样本身带来的收益和验证机制本身的收益混在一起,需要用匹配的正确基线重新核实。它也给出了一个具体的设计原则——选择性抑制超冲词元,而非均匀混合分布——用于设计更好的协作式验证方法。

本文术语

推测解码(Speculative Decoding) · 让小的草稿模型先提出候选词,大的目标模型并行验证,以加速大语言模型生成的技术
基于截断的验证(truncation-based verification) · 只要候选词落在按概率截断规则划定的允许集合内就直接接受的验证方式
协作式验证(collaborative verification) · 将草稿模型和目标模型的概率分布混合起来作为验证目标的方式
超冲(overshoot) · 草稿模型给某个词元分配的概率远高于目标模型给出的概率的现象
区块效率(Block Efficiency, BE) · 每一步平均被接受的词元数量,用来衡量解码速度

论文原文摘要（英文）

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality.

作者 · Tianyu Wang

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Tianyu Wang et al., arXiv:2607.26627, CC BY 4.0