K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

The Asymmetric Harms of LLM Compression

arXiv:2608.196702026-08-21

给AI模型瘦身压缩后,平均分数看起来没变,但常见知识反而流失更多,而且偏见会悄悄在特定群体中反向放大

研究团队对Llama-3.1-8B-Instruct、Qwen-3-8B和Gemma-2-9B-it三个语言模型应用了11种压缩方法(量化与剪枝),细致分析了知识保留能力、模型自信程度和社会偏见的变化。结果发现,压缩会让原本广为人知的头部知识相对损失更多,模型在给出错误答案时往往依然保持较高自信,而整体偏见分数看似稳定,实际上不同人口群体之间可能出现方向相反的巨大偏见变化。这说明仅凭准确率或困惑度这类平均指标,无法判断压缩后的模型是否真的安全可靠。

METAL MEDIA 解读图

给AI模型瘦身压缩后,平均分数看起来没变,但常见知识反而流失更多,而且偏见会悄悄在特定群体中反向放大

  1. 01对三个开源模型Llama-3.1-8B-Instruct、Qwen-3-8B、Gemma-2-9B-it应用了11种压缩方法,包括量化方法GPTQ、AWQ、OmniQuant、AQLM,以及剪枝方法WANDA、SparseGPT、ShortGPT等
  2. 02利用按知识流行度分为头部、中部、尾部的PopQA和Head-to-Tail数据集测试发现,压缩后头部知识虽然绝对准确率仍最高,但相对于模型自身基准表现而言,头部知识的保留比例反而比尾部知识损失更严重,出现了反转
  3. 03对于压缩后由正确变为错误的答案,模型常常仍保持中等偏高的自信程度(中位数约0.4到0.6),这种自信通常只在压缩强度极高、准确率已大幅崩溃之后才会明显下降
  4. 04在WinoBias和BBQ偏见测试基准上,整体偏见分数在许多设置下几乎没有变化,但具体人口子群体却出现了巨大且有时方向相反的变化,例如某一职业群体的变化达到负53.1个百分点,而整体变化却接近零
  5. 05这些隐藏效应在轻度到中度压缩下就已出现,当时常规指标看起来完全正常,因此作者主张在部署压缩模型前必须进行更细粒度的子群体级别评估
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 对三个开源模型Llama-3.1-8B-Instruct、Qwen-3-8B、Gemma-2-9B-it应用了11种压缩方法,包括量化方法GPTQ、AWQ、OmniQuant、AQLM,以及剪枝方法WANDA、SparseGPT、ShortGPT等
  2. 利用按知识流行度分为头部、中部、尾部的PopQA和Head-to-Tail数据集测试发现,压缩后头部知识虽然绝对准确率仍最高,但相对于模型自身基准表现而言,头部知识的保留比例反而比尾部知识损失更严重,出现了反转
  3. 对于压缩后由正确变为错误的答案,模型常常仍保持中等偏高的自信程度(中位数约0.4到0.6),这种自信通常只在压缩强度极高、准确率已大幅崩溃之后才会明显下降
  4. 在WinoBias和BBQ偏见测试基准上,整体偏见分数在许多设置下几乎没有变化,但具体人口子群体却出现了巨大且有时方向相反的变化,例如某一职业群体的变化达到负53.1个百分点,而整体变化却接近零
  5. 这些隐藏效应在轻度到中度压缩下就已出现,当时常规指标看起来完全正常,因此作者主张在部署压缩模型前必须进行更细粒度的子群体级别评估
Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8, 9, and 10 in Appendix A.2.1.
Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8, 9, and 10 in Appendix A.2.1.
Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive RSg indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14, 15, and 16 in Appendix A.2.1.
Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive RSg indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14, 15, and 16 in Appendix A.2.1.
Table 1: Quantization configurations.
ParameterGPTQAWQOmniQuantAQLM
Bit width2, 3, 42, 3, 42, 3, 42, 3, 4
Calibration datasetC4C4C4C4
Calibration samples128128128128
Calibration seed42424242
Calibration sequence length512512512512
Group size128128128–
Calibration splitvalidationvalidation––
Calibration source records4096–––
Calibration batch size11––
Symmetric quantizationTrueFalse––
Activation orderingTrue–––
Sequential quantizationTrue–––
Target modules–Linear––
Ignored modules–lm_head––
Activation bit width––16–
Optimization epochs––40 / 20 / 20–
Learnable weight clipping––True–
Learnable equivalent transformation––False–
Input group size–––8
Output group size–––1
Relative MSE tolerance–––0.01
Maximum fine-tuning epochs–––10
Activation offloading–––True
Resume enabled–––True
Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20, 21, and 22 in Appendix A.2.2.
Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20, 21, and 22 in Appendix A.2.2.
Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26, 27, and 28 in Appendix A.2.2.
Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26, 27, and 28 in Appendix A.2.2.
Table 2: Pruning configurations.
ParameterMagnitudeWANDASparseGPTShortGPTLayer Dropping
Compression granularityweightsweightsweightsdecoder blocksdecoder blocks
Compression levels30/50/70%30/50/70%30/50/70%5/10/15/20/25%5/10/15/20/25%
Semi-structured patterns–4:8, 2:44:8, 2:4––
Calibration dataset–C4C4C4C4
Calibration samples–32323232
Calibration sequence length–512512512256
Calibration split–validationvalidationvalidationvalidation
Target modulesattn./MLP linearattn./MLP linearattn./MLP lineardecoder blocksdecoder blocks
Excluded componentslm_headlm_headlm_headfirst/last blockfirst/last block
Selection criterionweight magnitudeweight–activation productHessian-based reconstructionblock influenceimportance + position
Activation-importance weight––––0.7
Position-prior weight––––0.35
Random seed––––13
Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38, 39, and 40 in Appendix A.2.3.
Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38, 39, and 40 in Appendix A.2.3.
Figure 6: Overall accuracy on PopQA under different compression methods and settings.
Figure 6: Overall accuracy on PopQA under different compression methods and settings.
Table 3: Perplexity of the three full-precision models and their compressed variants on WikiText-2.
CompressionMethodSettingPerplexity (PPL)
Llama-3.1- 8B-InstructQwen-3- 8BGemma-2- 9B-it
NoneFull precisionFP167.12539.588810.2117
QuantizationGPTQ2-bit1612.6539141.3107137.1019
3-bit10.035411.265212.3045
4-bit8.44599.951110.4903
AWQ2-bit94354.159716508.52549407.2547
3-bit9.493111.375011.8154
4-bit7.51929.994910.6473
OmniQuant2-bit671.255739.614735.2010
3-bit9.494911.707912.2236
4-bit7.572810.073710.5624
AQLM2-bit11.565913.068714.1902
3-bit11.320412.348412.2632
4-bit8.445710.280710.8105
Unstructured pruningMagnitude30% sparsity14.575611.552216.6274
50% sparsity177.720828.918965.7596
70% sparsity127104.4841138372.7513117527.1945
Wanda30% sparsity9.290111.032512.9266
50% sparsity12.738412.798417.2316
70% sparsity236.8802153.0605106.8877
SparseGPT30% sparsity9.587011.039513.9585
50% sparsity15.463713.993020.1264
70% sparsity272.6379862.9209129.0303
Semi-structured pruningWanda (N:M)4:818.026814.788619.6716
2:430.655118.415024.4520
SparseGPT (N:M)4:823.873216.913322.8913
2:443.447821.176733.7740
Structured pruningShortGPT5% blocks removed9.887215.147313.5355
10% blocks removed11.168335.141714.7864
15% blocks removed19.414143.593021.1945
Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.
Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.
Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.
Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.
Table 4: The ten largest absolute occupation-level bias changes on WinoBias among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes Δ​Bg and overall changes Δ​B are reported in percentage points.
RankGenderOccupationCompression ConfigurationBase 𝑩𝒈 (%)Comp. 𝑩𝒈 (%)𝚫​𝑩𝒈 (pp)𝚫​𝑩 (pp)
Llama-3.1-8B-Instruct
1FemaleSecretarySparseGPT N:M (2:4)71.918.8−53.1−2.2
3FemaleAuditorSparseGPT N:M (4:8)82.132.1−50.0+2.3
8MaleAnalystSparseGPT N:M (2:4)50.02.5−47.5−2.2
9FemaleClerkSparseGPT N:M (2:4)78.632.1−46.4−2.2
Qwen3-8B
2MaleSheriffMagnitude (50%)59.67.7−51.9−8.2
4FemaleClerkMagnitude (50%)67.917.9−50.0−8.2
5FemaleNurseShortGPT (20%)77.827.8−50.0−4.2
6MaleDeveloperSparseGPT N:M (2:4)71.921.9−50.0−4.0
7FemaleAttendantMagnitude (50%)61.812.7−49.1−8.2
10FemaleCounselorMagnitude (50%)66.119.6−46.4−8.2
Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.
Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.
Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.
Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.
Table 5: The ten largest absolute subgroup-level bias changes on BBQ among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes Δ​Bg and overall changes Δ​B are reported in percentage points.
RankCategorySubgroupCompression ConfigurationBase 𝑩𝒈 (%)Comp. 𝑩𝒈 (%)𝚫​𝑩𝒈 (pp)𝚫​𝑩 (pp)
Llama-3.1-8B-Instruct
8DisabilityPeople with cognitive disabilities or mental illnessSparseGPT N:M (4:8)40.659.4+18.8−1.7
9DisabilityPeople with cognitive disabilities or mental illnessWANDA N:M (2:4)40.659.4+18.8−3.1
Qwen3-8B
1DisabilityDown’s syndromeShortGPT (10%)37.575.0+37.5−1.2
2DisabilityDown’s syndromeShortGPT (15%)37.575.0+37.5−1.4
3DisabilityDown’s syndromeShortGPT (5%)37.562.5+25.0−1.1
4DisabilityPeople with cerebral palsySparseGPT (50%)56.381.3+25.0−1.0
10DisabilityDown’s syndromeMagnitude (30%)37.556.3+18.8−1.1
Gemma-2-9B-it
5DisabilityDown’s syndromeShortGPT (15%)50.075.0+25.0−0.5
6DisabilityDown’s syndromeShortGPT (25%)50.075.0+25.0−1.0
7NationalityItalianWANDA N:M (2:4)57.537.5−20.0+0.1
Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.
Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.
Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.
Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.

为什么重要

许多企业为降低部署成本而对大模型进行压缩,但这项研究表明,压缩模型可能在平均分数上看起来完好无损,实际上却悄悄丢失常见知识、对错误答案依然过度自信,并在性别、种族、残障等特定群体上产生不均衡的偏见变化。这提示实际部署压缩模型的团队,不能只看整体准确率或困惑度,还必须检查子群体层面的表现才能确保可靠性。

本文术语

  • 量化(Quantization) · 将模型权重或计算数值用更低精度(如从32位降到4位)存储,以节省内存和计算量的压缩方法
  • 剪枝(Pruning) · 删除模型中对输出贡献较小的权重、神经元或整层结构,使模型变小变快的压缩方法
  • 困惑度(Perplexity) · 衡量语言模型预测下一个词能力的常用整体指标,数值越低通常表示模型表现越好
  • 相对保留偏移(Relative Retention Shift) · 以百分点表示某一群体的准确率保留程度相对于模型整体保留程度高出或低出多少的指标
  • 期望校准误差(ECE) · 衡量模型所表达的自信程度与实际正确率是否吻合的指标,数值越低说明自信与准确性越一致

论文原文摘要（英文）

Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias. We find that compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge. Furthermore, compressed models often remain substantially confident in their incorrect answers on newly lost knowledge. Finally, we demonstrate that stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Together, these findings reveal asymmetric behavioral changes that aggregate performance measures fail to capture, highlighting the need for granular evaluation of compressed models before deployment.

作者 · Yuan Wu, Mairui Li, Lesia Semenova, Chudi Zhong

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Yuan Wu et al., arXiv:2608.19670, CC BY 4.0