Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

arXiv:2607.266272026-07-28

Popular speed tricks for LLM decoding quietly rewrite the output distribution and can hurt quality more than expected

Speculative decoding speeds up large language models by letting a small draft model propose tokens that a large target model verifies in parallel, and recent 'lossy verification' schemes loosen that verification to go even faster. This paper shows those schemes actually fall into just two underlying mechanisms, and both can degrade output quality more severely than reported once compared against the correct baseline. Code is released at the Fast-HSD GitHub repository.

METAL MEDIA explanatory visual

Why lossy verification schemes quietly distort output quality

Evidence statusMeasured results reported

  1. 1. Baseline: speculative decodingA small draft model q proposes tokens, a large target model p verifies them; lossless verification guarantees the final output distribution exactly matches p.
  2. 2. Two families of lossy verificationThe authors classify existing methods into truncation-based verification (accept if in an allowed set) and collaborative verification (blend p and q into a combined distribution).
  3. 3. Truncation pitfallAccepting anything inside the allowed set makes the output mirror the draft model q rather than the target p, and this distortion grows with task difficulty and under EAGLE-3's tree verification.
  4. 4. Collaborative fix: cap the overshootCapping only the tokens where the draft overshoots the target's probability preserves quality far better than uniformly mixing p and q across the whole vocabulary.
  5. 5. Correct comparison mattersApparent speedups from lossy verification should be checked against the matched truncation-sampling-only baseline on the target model, not against default decoding.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Speculative decoding accelerates LLM inference by having a lightweight draft model propose tokens that a larger target model verifies in parallel; recent 'lossy verification' methods relax the strict matching requirement to squeeze out more speed.
  2. The authors show that many seemingly different lossy verification methods reduce to just two families: truncation-based verification, which simply accepts any draft token that falls inside an allowed set defined by a truncation sampling rule, and collaborative verification, which blends the draft and target distributions.
  3. Across four benchmarks (MATH, MBPP+, INCLUDE, BFCL) using Qwen2.5-72B/0.5B and LLaMA-3.1 8B with EAGLE-3, truncation-based methods (SpecCascade, typical acceptance) consistently underperformed the matched baseline of simply applying the same truncation sampling directly to the target model, and this gap widened both with task difficulty and when using EAGLE-3's multi-draft tree verification.
  4. For collaborative verification, an ablation study found that selectively capping the 'overshoot' tokens -- where the draft model assigns much higher probability than the target model -- preserves quality far better than uniformly mixing the draft and target distributions, which is the mechanism CoS uses and which causes severe quality degradation.
Figure 1: Accuracy gap between the lossless baseline and truncation-based verification widens sharply with task difficulty, increasing from +0.38 pp on GSM8K to +6.67 pp on AIME. Here, the True baseline means the target model with min-p sampling, while the Wrong baseline indicates the target model w/o min-p sampling.
Figure 1: Accuracy gap between the lossless baseline and truncation-based verification widens sharply with task difficulty, increasing from +0.38 pp on GSM8K to +6.67 pp on AIME. Here, the True baseline means the target model with min-p sampling, while the Wrong baseline indicates the target model w/o min-p sampling.
Table 1: Ablation of the two mechanisms in lenience-based relaxation on MBPP+.
Variant𝝀BEPass@1 (%)
Adaptive interpolation0.29.1550.26
0.47.9356.61
0.66.8060.85
0.85.9566.14
Overshoot ceiling0.25.5975.93
0.45.5775.13
0.65.5475.66
0.85.5475.13
Figure 2: Comparison of distributions induced by collaborative and truncation-based methods. Tokens are sorted by decreasing target probability. Blue and pink bars denote the target distribution p and draft distribution q, respectively; orange denotes their overlap. The purple line shows the output distribution. Gray dashed lines indicate the ceiling.
Figure 2: Comparison of distributions induced by collaborative and truncation-based methods. Tokens are sorted by decreasing target probability. Blue and pink bars denote the target distribution p and draft distribution q, respectively; orange denotes their overlap. The purple line shows the output distribution. Gray dashed lines indicate the ceiling.
Table 2: Evaluation results using Qwen2.5-72B and Qwen2.5-0.5B pair under standard speculative decoding averaged over all hyperparameter settings. To ensure a fair comparison, each verification method is compared only with its matched truncation sampling baseline using the same allowed set. Bold denotes the better task performance within each matched pair. Gray denotes the SD baseline, which is not directly comparable.
MethodMATHMBPP+INCLUDEBFCL
BEDSAcc (%)BEDSPass@1 (%)BEDSAcc (%)BEDSAcc (%)
Standard SD (14)7.984.6776.475.474.0675.843.404.6368.188.736.8688.17
Min-p sampling (20)8.014.6476.515.533.8975.873.444.6267.798.856.9687.97
SpecCascade (19)8.234.6275.635.533.9075.883.584.6266.828.866.5288.30
η-sampling (12)7.984.6176.145.503.8975.853.364.6168.188.786.8288.17
Typical acceptance (4)8.394.6675.555.623.8975.803.584.6266.798.876.8388.93
Figure 3: Efficiency and task performance tradeoff for collaborative verification across four benchmarks. Each marker represents one hyperparameter setting, and lines connect settings in sweep order. The star and dashed reference lines denote the lossless speculative-decoding baseline. See Appendix C for complete results.
Figure 3: Efficiency and task performance tradeoff for collaborative verification across four benchmarks. Each marker represents one hyperparameter setting, and lines connect settings in sweep order. The star and dashed reference lines denote the lossless speculative-decoding baseline. See Appendix C for complete results.
Table 3: Evaluation results using LLaMA-3.1 8B and its official released pair under EAGLE-3 speculative decoding, averaged across the parameter grid for each method (T=0.7, block size =7). Bold denotes the better task performance within each matched pair. Gray denotes the EAGLE-3 baseline, which is not directly comparable.
MethodMATHMBPP+INCLUDEBFCL
BEDSAcc (%)BEDSPass@1 (%)BEDSAcc (%)BEDSAcc (%)
EAGLE-3 (16)3.76140.2173.204.70167.8359.300.6745.7235.502.5785.2786.00
Min-p sampling (20)3.98143.2276.884.90168.5761.800.5645.8037.002.5884.2686.60
SpecCascade (19)4.22152.7476.165.00175.9160.740.7848.1333.272.5985.2085.40
η-sampling (12)3.93140.0576.124.87165.6162.170.5545.3336.732.5879.7786.30
Typical acceptance (4)4.61162.7369.845.20180.1056.881.0855.5427.912.6485.6181.40
Figure 4: Net change in Δ​BE under different draft–target alignment ratios for Min-p and η-sampling. Each triplet reports the ratios of matching, partially overlapping, and unrelated candidate tokens, respectively.
Figure 4: Net change in Δ​BE under different draft–target alignment ratios for Min-p and η-sampling. Each triplet reports the ratios of matching, partially overlapping, and unrelated candidate tokens, respectively.
Table 4: Verification method results across the four benchmarks (MATH, MBPP+, INCLUDE, BFCL). For each method we sweep its characteristic hyperparameter (Param) and report block efficiency (BE), decoding speed (DS), and accuracy / Pass@1; all entries are mean ± std over three seeds.
MethodParamMATHMBPP+INCLUDEBFCL
BEDSAcc (%)BEDSPass@1 (%)BEDSAcc (%)BEDSAcc (%)
SD Baseline 14–7.98±0.024.67±0.0276.47±1.535.47±0.064.06±0.0375.84±0.403.40±0.034.63±0.0168.18±0.008.73±0.016.86±0.0088.17±0.58
Min-p Smpl. + tokenwise SpD0.17.94±0.024.65±0.0276.27±1.535.42±0.083.89±0.1475.57±0.403.34±0.044.63±0.0168.79±0.698.73±0.016.97±0.1888.17±0.58
0.37.99±0.054.64±0.0176.67±0.835.56±0.033.90±0.1476.19±0.463.42±0.044.62±0.0168.18±2.088.85±0.016.94±0.2188.33±1.04
0.58.03±0.024.64±0.0176.07±1.175.45±0.073.89±0.1475.57±0.313.44±0.024.61±0.0268.48±1.468.88±0.026.97±0.1987.67±0.29
0.78.08±0.044.64±0.0176.27±0.425.59±0.023.88±0.1376.01±0.313.47±0.054.63±0.0166.52±1.148.89±0.026.96±0.1987.67±0.29
0.98.08±0.104.65±0.0177.27±1.275.62±0.043.88±0.1476.01±0.153.47±0.044.62±0.0166.97±1.318.89±0.016.97±0.1888.00±0.00
Cascade 50.18.46±0.064.53±0.2276.87±0.465.64±0.033.89±0.1476.46±0.263.57±0.044.63±0.0164.85±1.148.87±0.046.83±0.0589.00±0.50
0.38.36±0.044.65±0.0175.47±0.705.59±0.043.89±0.1475.40±0.263.50±0.034.60±0.0166.67±3.448.89±0.016.81±0.0288.67±0.29
0.58.12±0.054.66±0.0174.87±0.425.51±0.033.90±0.1274.87±0.703.40±0.084.61±0.0268.33±3.038.85±0.036.88±0.0388.00±0.00
0.78.01±0.064.66±0.0175.47±0.705.48±0.013.90±0.1476.46±0.533.30±0.044.60±0.0168.33±1.058.84±0.036.82±0.0387.67±0.29
0.97.84±0.074.66±0.0175.47±1.405.41±0.013.89±0.1476.19±0.263.25±0.034.62±0.0165.91±0.458.84±0.035.28±0.0488.17±0.58
η Smpl. + tokenwise SpD0.057.92±0.044.63±0.0176.67±1.855.42±0.083.89±0.1475.57±0.403.34±0.054.61±0.0267.73±1.828.73±0.016.95±0.1888.17±0.58
0.107.94±0.034.64±0.0176.47±1.035.49±0.063.88±0.1576.54±0.613.31±0.024.62±0.0268.79±0.268.74±0.016.82±0.0288.17±0.58
0.157.96±0.034.64±0.0275.87±2.215.44±0.023.89±0.1576.01±0.313.39±0.054.60±0.0367.27±0.458.77±0.016.79±0.0588.17±0.58
0.208.11±0.294.51±0.2176.27±0.705.57±0.053.88±0.1276.01±0.153.42±0.054.61±0.0168.33±1.848.80±0.076.80±0.0288.17±0.58
0.258.03±0.064.50±0.2175.40±1.605.57±0.043.87±0.1475.13±0.263.41±0.024.63±0.0368.79±1.608.88±0.056.72±0.0788.17±0.58
Medusa 40.058.49±0.014.67±1.9376.60±2.085.64±0.033.89±0.1476.46±1.063.58±0.034.62±0.0166.52±1.318.87±0.046.84±0.0289.00±0.50
0.108.47±0.054.66±0.0075.20±1.115.64±0.033.89±0.1376.46±1.063.63±0.034.61±0.0166.52±1.058.87±0.046.85±0.0189.00±0.50
0.158.21±0.244.65±0.0375.73±2.125.66±0.033.88±0.1475.66±0.533.59±0.014.62±0.0266.06±1.318.87±0.046.85±0.0389.00±0.50
0.208.39±0.014.65±0.0074.67±1.555.63±0.053.90±0.1475.40±0.263.54±0.034.60±0.0168.79±0.698.87±0.046.84±0.0289.00±0.50
0.258.30±0.054.67±0.0175.53±0.315.55±0.083.90±0.1475.04±0.153.46±0.014.63±0.0166.06±0.268.89±0.016.78±0.0488.67±0.29
Lenience-based relaxation0.28.49±0.024.68±0.0574.47±0.585.65±0.034.06±0.0275.13±0.263.59±0.044.52±0.0467.42±1.398.87±0.016.90±0.0288.67±0.29
0.48.37±0.084.66±0.1275.60±2.655.61±0.024.06±0.0275.49±0.673.55±0.064.53±0.0967.88±1.318.85±0.046.87±0.0188.00±0.50
0.68.26±0.054.66±0.1478.00±1.735.52±0.074.06±0.0375.57±0.403.48±0.024.50±0.1868.64±1.828.83±0.026.84±0.0288.17±0.29
0.88.22±0.074.65±0.2376.47±1.155.52±0.044.06±0.0275.75±0.403.38±0.034.49±0.1168.33±0.268.81±0.056.87±0.0287.67±0.29
CoS 90.28.33±0.004.68±0.0171.53±2.126.05±0.134.10±0.0171.43±1.063.89±0.044.61±0.0264.55±1.988.76±0.136.79±0.0470.67±3.40
0.48.80±0.044.67±0.0060.40±2.847.14±0.094.13±0.0161.20±3.294.69±0.074.58±0.0353.79±2.108.84±0.156.69±0.0358.00±4.50
0.69.42±0.014.67±0.0154.80±0.727.86±0.024.20±0.0259.44±0.855.99±0.034.53±0.0148.03±1.899.29±0.166.69±0.0343.83±3.79
0.810.14±0.114.65±0.0146.00±2.129.26±0.044.26±0.0054.06±0.318.02±0.054.49±0.0137.12±2.0510.26±0.076.75±0.0143.67±1.89
Figure 5: Block efficiency of truncation-based methods across hyperparameter settings on four benchmarks. Top row: Min-p sampling and SpecCascade as pbase varies. Bottom row: η-sampling and typical acceptance as ϵ varies. Lines show the mean and shaded regions indicate one standard deviation. The dashed line denotes the lossless speculative-decoding baseline under the default Qwen2.5 sampling parameters.
Figure 5: Block efficiency of truncation-based methods across hyperparameter settings on four benchmarks. Top row: Min-p sampling and SpecCascade as pbase varies. Bottom row: η-sampling and typical acceptance as ϵ varies. Lines show the mean and shaded regions indicate one standard deviation. The dashed line denotes the lossless speculative-decoding baseline under the default Qwen2.5 sampling parameters.
Table 5: EAGLE-3 speculative decoding results across four benchmarks (LLaMA-3.1 8B, temperature =0.7, block size=7). Bold marks the best value per column across all methods; underline marks the second best.
MATHMBPP+INCLUDEBFCL
MethodParamBEDSAcc (%)BEDSPass@1 (%)BEDSAcc (%)BEDSAcc (%)
Baseline—3.76140.2173.204.70167.8359.300.6745.7235.502.5785.2786.00
Lenience 140.24.49161.2971.205.17181.9561.110.8650.6130.912.6387.3183.50
0.44.34157.0376.205.02177.1359.790.8048.9332.732.6186.1485.50
0.64.19152.8576.004.96175.4359.520.7347.0334.552.5985.7786.00
0.84.07149.3975.204.92174.2559.790.7246.7333.642.5985.5684.00
SpecCascade 190.14.47159.8373.005.16180.8159.790.9251.7630.912.6285.8284.50
0.34.28154.4277.405.03176.7961.380.7948.2631.822.5985.1884.50
0.54.18151.5677.604.99175.6660.850.7647.4331.822.5885.0386.50
0.74.12149.7176.204.93173.7061.380.7346.7436.822.5884.9985.50
0.94.06148.2076.604.89172.6060.320.7246.4535.002.5885.0086.00
Min-p Smpl. + SD0.13.89140.7275.204.84166.7961.900.5445.3440.002.5884.4586.00
0.33.96142.8577.604.89168.2960.320.5545.6536.822.5884.1386.00
0.53.98143.4277.404.91168.9860.850.5645.9135.912.5884.0986.50
0.74.00143.8877.404.92169.2362.430.5645.9635.912.5984.2187.50
0.94.05145.2476.804.93169.5863.490.5746.1336.362.5984.4287.00
η Sampl. + SD0.053.85137.9275.204.84164.7959.790.5545.4235.002.5883.6387.50
0.103.90139.3377.004.86165.3261.110.5445.0641.822.5983.9286.00
0.153.93140.2177.204.88165.9561.900.5445.1335.002.5983.9186.50
0.203.96140.9776.004.89166.2364.290.5445.1434.092.5783.5485.00
0.253.99141.8375.204.88165.7763.760.5745.9237.732.5863.8586.50
Typical Sampling0.054.82168.7766.005.29182.8655.291.2459.8029.552.6686.1178.00
0.104.66164.1968.205.21180.5456.081.1557.2029.552.6585.8579.50
0.154.58161.8672.205.16178.8757.411.0554.7625.912.6485.6583.00
0.204.54160.6569.805.16178.9458.470.9852.7327.732.6385.4083.50
0.254.46158.1773.005.17179.3157.141.0053.2126.822.6285.0583.00
Figure 6: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on four benchmarks. Each point represents a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling.
Figure 6: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on four benchmarks. Each point represents a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling.
Table 6: Effect of truncation and overshoot capping on MBPP+, averaged over ℓ∈{0.2,0.4,0.6,0.8}. pbase=0.5 is used in 𝒜Θ.
Yielded ruleAvg. BEAvg. Pass@1 (%)
P⁡(generate​x)=p⁡(x)5.4275.66
P⁡(generate​x)={q⁡(x),x∈𝒜Θ,0,otherwise,5.51 (+1.7%)74.87 (-1.0%)
P⁡(generate​x)={Δ​p​(x)+(1−Δ)​q​(x),q⁡(x)≤p⁡(x),q⁡(x),p⁡(x)<q⁡(x)≤p⁡(x)/ℓ,p⁡(x)/ℓ,q⁡(x)≥p⁡(x)/ℓ,5.53 (+2.0%)75.22 (-0.6%)
P⁡(generate​x)={q⁡(x),q⁡(x)≤p⁡(x)/ℓ,p⁡(x)/ℓ,q⁡(x)≥p⁡(x)/ℓ,5.58 (+3.0%)75.33 (-0.4%)
P⁡(generate​x)={q⁡(x),x∈𝒜Θ,min⁡{q⁡(x),p⁡(x)/ℓ},otherwise5.62 (+3.7%)75.66 (-0.0%)
Figure 7: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on three benchmarks. Each point shows a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling. Detailed results can be found in Appendix C.
Figure 7: Efficiency and task performance tradeoff for truncation methods across hyperparameter settings on three benchmarks. Each point shows a method’s BE and corresponding accuracy or Pass@1, under certain hyper-parameter. Baseline refers to the default decoding configuration without min-p or η sampling. Detailed results can be found in Appendix C.
Table 7: Gap of truncation-based verification relative to its matched truncation sampling + SD baseline (Δ= lossy − lossless), without EAGLE-3 (single-draft SD, Table 2) and with EAGLE-3 (Table 3). EAGLE-3 amplifies the average task-performance deficit by roughly 4× (SpecCascade) and 20× (typical acceptance).
Matched pairFrameworkMATHMBPP+INCLUDEBFCLAvg.
ΔBEΔAccΔBEΔAccΔBEΔAccΔBEΔAccΔBEΔAcc
SpecCascade − Min-p samplingSingle-draft SD+0.22−0.88+0.00+0.01+0.14−0.97+0.01+0.33+0.09−0.38
EAGLE-3+0.24−0.72+0.10−1.06+0.22−3.73+0.01−1.20+0.14−1.68
Typical acceptance − η-samplingSingle-draft SD+0.41−0.59+0.12−0.05+0.22−1.39+0.09+0.76+0.21−0.32
EAGLE-3+0.68−6.28+0.33−5.29+0.53−8.82+0.06−4.90+0.40−6.32
Figure 8: Two views of lenience-based relaxation. Fig.8(a) illustrates how the interpolation strength adapts to the draft–target divergence on a per-token basis. Fig.8(b) contrasts a global linear mixture with a pointwise ceiling at p/ℓ, highlighting the asymmetric treatment of overshoot.
Figure 8: Two views of lenience-based relaxation. Fig.8(a) illustrates how the interpolation strength adapts to the draft–target divergence on a per-token basis. Fig.8(b) contrasts a global linear mixture with a pointwise ceiling at p/ℓ, highlighting the asymmetric treatment of overshoot.

Findings

  • Under standard speculative decoding, truncation-based methods (SpecCascade, typical acceptance) underperformed their matched truncation-sampling-only baseline on every one of the four benchmarks, with per-benchmark gaps up to 1.4 points.
  • Under EAGLE-3's multi-draft tree verification, this gap widened substantially: SpecCascade's average deficit grew from -0.38 to -1.68 points and typical acceptance's grew from -0.32 to -6.32 points, reaching as much as -8.8 points on INCLUDE.
  • The accuracy gap between the lossless baseline and truncation-based verification widened from +0.38 percentage points on the easy GSM8K benchmark to +6.67 percentage points on the hard AIME benchmark.
  • An ablation on MBPP+ showed that the effectiveness of lenience-based relaxation comes from capping the overshoot region at a ceiling (p/l), not from the adaptive interpolation in the underestimation region; the ceiling alone matched lossless verification's task performance while still improving efficiency.
  • A combined rule that gates acceptance with the min-p allowed set and applies the overshoot cap outside it achieved a 3.7% gain in block efficiency over lossless speculative decoding while matching its Pass@1 exactly.
(b) Overshoot ceiling.
(b) Overshoot ceiling.

Where it can be used

  • Teams deploying speculative decoding to speed up LLM inference can use this framework to check whether a lossy verification method's reported speedup actually beats simply applying the same truncation sampling directly to the target model.
  • For tasks where precision matters, such as code generation, math reasoning, multilingual understanding, or tool calling, this analysis helps identify which verification schemes carry lower risk of quality degradation.
  • Teams considering combining lossy verification with tree-based multi-draft systems like EAGLE-3 should weigh that quality degradation can be much larger than under standard single-draft speculative decoding.
  • Designers of new collaborative verification schemes can consider selectively suppressing overshoot tokens rather than uniformly blending draft and target distributions.

Limits and open work

  • Experiments are limited to the Qwen2.5 and Llama-3.1 model families with a fixed target-draft pairing, so it remains unverified whether the magnitude of the truncation pitfall and the overshoot principle transfers to other architectures, scales, or draft-target ratios.
  • Evaluation focuses on reasoning, code, multilingual, and function-calling benchmarks (MATH, MBPP+, INCLUDE, BFCL); open-ended generation and dialogue settings, where quality is harder to measure automatically, are not covered.
  • The theoretical analysis assumes the specific truncation and collaborative rules formalized in the paper and may not directly extend to future lossy verification schemes not covered here.
  • Block efficiency is reported as a hardware-agnostic proxy for speed, but actual wall-clock speedups depend on hardware and serving stack, so the real speed-quality trade-off can vary by deployment environment.

Why it matters

For anyone deploying speculative decoding to speed up LLM inference, this work warns that reported speedups from lossy verification schemes may be conflated with gains from the underlying truncation sampling itself, and offers a concrete way to check whether a given method is actually worth the quality trade-off. It also gives a concrete design principle -- suppress overshoot tokens rather than blend distributions uniformly -- for building better collaborative verification schemes.

Terms in this paper

  • Speculative Decoding · A technique where a small draft model proposes upcoming tokens and a large target model verifies them in parallel to speed up generation
  • truncation-based verification · A verification rule that accepts any draft token as long as it falls inside a probability-cutoff allowed set
  • collaborative verification · A verification rule that blends the draft and target model's probability distributions rather than matching the target exactly
  • overshoot · When the draft model assigns a token much higher probability than the target model does
  • Block Efficiency (BE) · The average number of tokens accepted per decoding step, used as a speed metric

Original abstract (English)

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality.

Authors · Tianyu Wang

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Tianyu Wang et al., arXiv:2607.26627, CC BY 4.0