컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

AI 모델을 가볍게 압축하면 평균 점수는 그대로여도, 흔한 지식이 오히려 더 많이 사라지고 편향은 특정 집단에서 몰래 커진다

arXiv:2608.196702026-08-21

The Asymmetric Harms of LLM Compression

AI 모델을 가볍게 압축하면 평균 점수는 그대로여도, 흔한 지식이 오히려 더 많이 사라지고 편향은 특정 집단에서 몰래 커진다

연구팀은 Llama, Qwen, Gemma 세 개의 언어모델에 11가지 압축 기법(양자화·가지치기)을 적용해 지식 유지력, 자신감, 사회적 편향을 정밀 분석했다. 그 결과 압축은 흔히 알려진 지식(헤드)을 상대적으로 더 많이 잃게 만들고, 모델은 틀린 답을 내면서도 여전히 자신감을 유지하며, 전체 편향 점수는 안정적이어도 성별·인종 등 하위 집단에서는 정반대 방향의 큰 변화가 숨어 있었다. 즉 평균 정확도나 복잡도(perplexity) 같은 기존 지표만으로는 압축된 모델의 실제 위험을 파악할 수 없다는 것이다.

METAL MEDIA 해설 도표

AI 모델을 가볍게 압축하면 평균 점수는 그대로여도, 흔한 지식이 오히려 더 많이 사라지고 편향은 특정 집단에서 몰래 커진다

  1. 01Llama-3.1-8B-Instruct, Qwen-3-8B, Gemma-2-9B-it 세 모델에 양자화(GPTQ, AWQ, OmniQuant, AQLM)와 가지치기(WANDA, SparseGPT, ShortGPT 등) 11가지 기법을 적용해 비교했다
  2. 02PopQA와 Head-to-Tail 데이터셋으로 지식을 유명도(헤드·미들·테일)에 따라 나누어 정확도 유지율을 측정했더니, 절대 정확도는 헤드가 여전히 높지만 기준 모델 대비 상대적 유지율은 오히려 테일 지식이 더 잘 보존되고 헤드 지식이 더 많이 깎였다
  3. 03압축으로 인해 새로 틀리게 된 답변에서도 모델은 중간~높은 확신(0.4~0.6 수준)을 유지하는 경우가 많았고, 이 확신은 정확도가 이미 무너진 뒤에야 급락하는 경우가 대부분이었다
  4. 04WinoBias, BBQ 편향 벤치마크에서 전체 편향 점수는 거의 변화가 없어도 남성·여성 등 하위 집단별로는 최대 수십 퍼센트포인트(예: 특정 직업군에서 -53.1퍼센트포인트)에 달하는 상반된 변화가 발생했다
  5. 05경미하거나 중간 수준의 압축에서도 이런 숨겨진 불균형이 나타났기 때문에, 평균 지표만 보고 압축 모델을 안전하다고 판단해서는 안 되며 세부 집단별 검증이 필요하다는 결론을 내렸다
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. Llama-3.1-8B-Instruct, Qwen-3-8B, Gemma-2-9B-it 세 모델에 양자화(GPTQ, AWQ, OmniQuant, AQLM)와 가지치기(WANDA, SparseGPT, ShortGPT 등) 11가지 기법을 적용해 비교했다
  2. PopQA와 Head-to-Tail 데이터셋으로 지식을 유명도(헤드·미들·테일)에 따라 나누어 정확도 유지율을 측정했더니, 절대 정확도는 헤드가 여전히 높지만 기준 모델 대비 상대적 유지율은 오히려 테일 지식이 더 잘 보존되고 헤드 지식이 더 많이 깎였다
  3. 압축으로 인해 새로 틀리게 된 답변에서도 모델은 중간~높은 확신(0.4~0.6 수준)을 유지하는 경우가 많았고, 이 확신은 정확도가 이미 무너진 뒤에야 급락하는 경우가 대부분이었다
  4. WinoBias, BBQ 편향 벤치마크에서 전체 편향 점수는 거의 변화가 없어도 남성·여성 등 하위 집단별로는 최대 수십 퍼센트포인트(예: 특정 직업군에서 -53.1퍼센트포인트)에 달하는 상반된 변화가 발생했다
  5. 경미하거나 중간 수준의 압축에서도 이런 숨겨진 불균형이 나타났기 때문에, 평균 지표만 보고 압축 모델을 안전하다고 판단해서는 안 되며 세부 집단별 검증이 필요하다는 결론을 내렸다
Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8, 9, and 10 in Appendix A.2.1.
Figure 1: Accuracy across different popularity groups on PopQA under different compression settings. Results for all methods and models are provided in Figures 8, 9, and 10 in Appendix A.2.1.
Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive RSg indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14, 15, and 16 in Appendix A.2.1.
Figure 2: Relative retention shift across PopQA popularity groups under different compression settings, reported in percentage points. Negative and positive RSg indicate lower and higher relative retention than the dataset overall, respectively. Results for all methods and models are provided in Figures 14, 15, and 16 in Appendix A.2.1.
Table 1: Quantization configurations.
ParameterGPTQAWQOmniQuantAQLM
Bit width2, 3, 42, 3, 42, 3, 42, 3, 4
Calibration datasetC4C4C4C4
Calibration samples128128128128
Calibration seed42424242
Calibration sequence length512512512512
Group size128128128–
Calibration splitvalidationvalidation––
Calibration source records4096–––
Calibration batch size11––
Symmetric quantizationTrueFalse––
Activation orderingTrue–––
Sequential quantizationTrue–––
Target modules–Linear––
Ignored modules–lm_head––
Activation bit width––16–
Optimization epochs––40 / 20 / 20–
Learnable weight clipping––True–
Learnable equivalent transformation––False–
Input group size–––8
Output group size–––1
Relative MSE tolerance–––0.01
Maximum fine-tuning epochs–––10
Activation offloading–––True
Resume enabled–––True
Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20, 21, and 22 in Appendix A.2.2.
Figure 3: Knowledge-loss rate across popularity groups on PopQA under different compression settings. Higher values indicate that a larger proportion of the knowledge correctly answered by the base model is incorrect after compression. Results for all methods and models are provided in Figures 20, 21, and 22 in Appendix A.2.2.
Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26, 27, and 28 in Appendix A.2.2.
Figure 4: Confidence on lost knowledge across popularity groups on PopQA under different compression settings. Each dot represents the median confidence; the thick vertical bar shows the 25th–75th percentile range, and the thin vertical bar shows the 5th–95th percentile range. Colors distinguish tail, middle, and head knowledge. Results for all methods and models are provided in Figures 26, 27, and 28 in Appendix A.2.2.
Table 2: Pruning configurations.
ParameterMagnitudeWANDASparseGPTShortGPTLayer Dropping
Compression granularityweightsweightsweightsdecoder blocksdecoder blocks
Compression levels30/50/70%30/50/70%30/50/70%5/10/15/20/25%5/10/15/20/25%
Semi-structured patterns–4:8, 2:44:8, 2:4––
Calibration dataset–C4C4C4C4
Calibration samples–32323232
Calibration sequence length–512512512256
Calibration split–validationvalidationvalidationvalidation
Target modulesattn./MLP linearattn./MLP linearattn./MLP lineardecoder blocksdecoder blocks
Excluded componentslm_headlm_headlm_headfirst/last blockfirst/last block
Selection criterionweight magnitudeweight–activation productHessian-based reconstructionblock influenceimportance + position
Activation-importance weight––––0.7
Position-prior weight––––0.35
Random seed––––13
Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38, 39, and 40 in Appendix A.2.3.
Figure 5: Bias changes on WinoBias under different compression settings, reported in percentage points. Blue, orange, and black bars denote changes for male, female, and all examples, respectively, and error bars denote 95% confidence intervals. Positive values indicate increased stereotypical preference, negative values indicate decreased preference, and the dashed line marks no change from the base model. Results for all methods and models are provided in Figures 38, 39, and 40 in Appendix A.2.3.
Figure 6: Overall accuracy on PopQA under different compression methods and settings.
Figure 6: Overall accuracy on PopQA under different compression methods and settings.
Table 3: Perplexity of the three full-precision models and their compressed variants on WikiText-2.
CompressionMethodSettingPerplexity (PPL)
Llama-3.1- 8B-InstructQwen-3- 8BGemma-2- 9B-it
NoneFull precisionFP167.12539.588810.2117
QuantizationGPTQ2-bit1612.6539141.3107137.1019
3-bit10.035411.265212.3045
4-bit8.44599.951110.4903
AWQ2-bit94354.159716508.52549407.2547
3-bit9.493111.375011.8154
4-bit7.51929.994910.6473
OmniQuant2-bit671.255739.614735.2010
3-bit9.494911.707912.2236
4-bit7.572810.073710.5624
AQLM2-bit11.565913.068714.1902
3-bit11.320412.348412.2632
4-bit8.445710.280710.8105
Unstructured pruningMagnitude30% sparsity14.575611.552216.6274
50% sparsity177.720828.918965.7596
70% sparsity127104.4841138372.7513117527.1945
Wanda30% sparsity9.290111.032512.9266
50% sparsity12.738412.798417.2316
70% sparsity236.8802153.0605106.8877
SparseGPT30% sparsity9.587011.039513.9585
50% sparsity15.463713.993020.1264
70% sparsity272.6379862.9209129.0303
Semi-structured pruningWanda (N:M)4:818.026814.788619.6716
2:430.655118.415024.4520
SparseGPT (N:M)4:823.873216.913322.8913
2:443.447821.176733.7740
Structured pruningShortGPT5% blocks removed9.887215.147313.5355
10% blocks removed11.168335.141714.7864
15% blocks removed19.414143.593021.1945
Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.
Figure 7: Overall accuracy on Head-to-Tail under different compression methods and settings.
Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.
Figure 8: Accuracy across different popularity groups on PopQA for Llama-3.1-8B-Instruct under additional compression methods and settings.
Table 4: The ten largest absolute occupation-level bias changes on WinoBias among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes Δ​Bg and overall changes Δ​B are reported in percentage points.
RankGenderOccupationCompression ConfigurationBase 𝑩𝒈 (%)Comp. 𝑩𝒈 (%)𝚫​𝑩𝒈 (pp)𝚫​𝑩 (pp)
Llama-3.1-8B-Instruct
1FemaleSecretarySparseGPT N:M (2:4)71.918.8−53.1−2.2
3FemaleAuditorSparseGPT N:M (4:8)82.132.1−50.0+2.3
8MaleAnalystSparseGPT N:M (2:4)50.02.5−47.5−2.2
9FemaleClerkSparseGPT N:M (2:4)78.632.1−46.4−2.2
Qwen3-8B
2MaleSheriffMagnitude (50%)59.67.7−51.9−8.2
4FemaleClerkMagnitude (50%)67.917.9−50.0−8.2
5FemaleNurseShortGPT (20%)77.827.8−50.0−4.2
6MaleDeveloperSparseGPT N:M (2:4)71.921.9−50.0−4.0
7FemaleAttendantMagnitude (50%)61.812.7−49.1−8.2
10FemaleCounselorMagnitude (50%)66.119.6−46.4−8.2
Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.
Figure 9: Accuracy across different popularity groups on PopQA for Qwen3-8B under additional compression methods and settings.
Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.
Figure 10: Accuracy across different popularity groups on PopQA for Gemma-2-9B under additional compression methods and settings.
Table 5: The ten largest absolute subgroup-level bias changes on BBQ among the retained compression configurations. Base and compressed subgroup bias scores are reported as percentages, while subgroup changes Δ​Bg and overall changes Δ​B are reported in percentage points.
RankCategorySubgroupCompression ConfigurationBase 𝑩𝒈 (%)Comp. 𝑩𝒈 (%)𝚫​𝑩𝒈 (pp)𝚫​𝑩 (pp)
Llama-3.1-8B-Instruct
8DisabilityPeople with cognitive disabilities or mental illnessSparseGPT N:M (4:8)40.659.4+18.8−1.7
9DisabilityPeople with cognitive disabilities or mental illnessWANDA N:M (2:4)40.659.4+18.8−3.1
Qwen3-8B
1DisabilityDown’s syndromeShortGPT (10%)37.575.0+37.5−1.2
2DisabilityDown’s syndromeShortGPT (15%)37.575.0+37.5−1.4
3DisabilityDown’s syndromeShortGPT (5%)37.562.5+25.0−1.1
4DisabilityPeople with cerebral palsySparseGPT (50%)56.381.3+25.0−1.0
10DisabilityDown’s syndromeMagnitude (30%)37.556.3+18.8−1.1
Gemma-2-9B-it
5DisabilityDown’s syndromeShortGPT (15%)50.075.0+25.0−0.5
6DisabilityDown’s syndromeShortGPT (25%)50.075.0+25.0−1.0
7NationalityItalianWANDA N:M (2:4)57.537.5−20.0+0.1
Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.
Figure 11: Accuracy across different popularity groups on Head-to-Tail for Llama-3.1-8B-Instruct under different compression methods and settings.
Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.
Figure 12: Accuracy across different popularity groups on Head-to-Tail for Qwen3-8B under different compression methods and settings.

왜 중요한가

많은 기업과 서비스가 비용 절감을 위해 모델을 가볍게 만드는 압축 기법을 쓰는데, 이 연구는 평균 성능이 멀쩡해 보여도 실제로는 상식적으로 흔한 지식이 더 취약해지고, 모델이 틀린 답에도 자신 있게 말하며, 특정 성별·인종·질환 집단에 대한 편향이 몰래 커질 수 있음을 보여준다. 따라서 압축 모델을 배포하기 전에는 전체 점수뿐 아니라 하위 집단별 세부 평가가 반드시 필요하다는 실무적 경고를 준다.

이 논문의 용어

  • 양자화(Quantization) · 모델의 가중치나 계산값을 더 낮은 정밀도(예: 32비트→4비트)로 저장해 용량과 연산량을 줄이는 압축 기법
  • 가지치기(Pruning) · 모델에서 출력에 영향을 적게 주는 가중치, 뉴런, 층 등을 제거해 크기를 줄이는 압축 기법
  • 혼란도(Perplexity) · 언어모델이 다음 단어를 얼마나 잘 예측하는지를 나타내는 대표적 평균 성능 지표, 낮을수록 좋음
  • 상대적 유지율 변화(Relative Retention Shift) · 압축 후 특정 집단의 정확도 유지 정도가 전체 평균 대비 얼마나 더 좋거나 나쁜지를 퍼센트포인트로 나타낸 지표
  • 기대 보정 오차(ECE) · 모델이 표현한 확신 수준이 실제 정답률과 얼마나 잘 맞는지를 측정하는 지표, 낮을수록 확신과 정확도가 잘 일치함을 의미

저자 · Yuan Wu, Mairui Li, Lesia Semenova, Chudi Zhong

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: Yuan Wu et al., arXiv:2608.19670, CC BY 4.0