컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

METAL MEDIA

구글의 DiffusionGemma는 한 번에 256개 토큰 블록을 동시에 다듬어서 기존 AR 모델보다 훨씬 빠르게 텍스트를 생성하는 실험적 개방형 언어모델이다

arXiv:2608.001462026-07-30

DiffusionGemma Technical Report

구글의 DiffusionGemma는 한 번에 256개 토큰 블록을 동시에 다듬어서 기존 AR 모델보다 훨씬 빠르게 텍스트를 생성하는 실험적 개방형 언어모델이다

DiffusionGemma는 한 토큰씩 순서대로 생성하는 기존 방식(자기회귀, AR) 대신, 256개 토큰 묶음을 동시에 반복적으로 다듬는 이산 확산(discrete diffusion) 방식으로 텍스트를 만든다. 처음부터 학습시키지 않고, 활성 파라미터 38억 개, 전체 파라미터 252억 개인 혼합전문가(MoE) 구조의 Gemma 4 모델을 미세조정해서 만들었으며, 원래 AR 모델 학습 토큰 예산의 10% 미만만 사용했다. 평가 전반에서 평균적으로 한 번의 순전파마다 약 20개 토큰을 만들어내고, 단일 H100 GPU에서 초당 약 1,500개 토큰을 생성해 최신 추측 디코딩을 쓰는 AR 모델보다도 훨씬 빠른 속도를 보였다.

METAL MEDIA 해설 도표

DiffusionGemma가 텍스트를 만드는 두 단계 흐름

증거 상태측정 결과가 보고됨

  1. 시작점: Gemma 4 AR 모델활성 38억, 전체 252억 파라미터의 혼합전문가(MoE) AR 모델 가중치에서 출발, 처음부터 새로 학습하지 않음
  2. 1단계 SFT256토큰 캔버스에 대해 양방향 주의로 노이즈를 제거하도록 지도 미세조정, AR 모델 학습 토큰 예산의 10% 미만 사용
  3. 2단계 SD·RL샘플러 증류와 강화학습을 결합해 보상 기반 품질 향상과 디노이징 스텝 수 압축을 동시에 학습
  4. 추론: 블록 단위 확산 생성무작위 노이즈 캔버스에서 시작해 엔트로피 기반 샘플러와 적응형 정지로 평균 약 12스텝만에 256토큰 블록을 완성, 완성된 블록은 KV 캐시에 이어붙임
  5. 결과: 속도-품질 파레토 프론티어H100 한 대에서 초당 약 1,500토큰, 순전파당 약 20토큰으로 최신 추측 디코딩 AR 모델보다 빠른 지점을 새로 개척
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

  1. 기존 AR 언어모델은 한 번에 토큰 하나씩만 만들어서, 특히 요청이 적을 때는 GPU 연산 능력을 다 못 쓰고 메모리 전송 속도에 발목 잡힌다(메모리 병목).
  2. DiffusionGemma는 이 문제를 이산 확산(discrete diffusion)으로 우회한다. 무작위 노이즈 상태의 256토큰 캔버스에서 시작해, 양방향 주의(attention)로 캔버스 전체를 동시에 조금씩 정제하며 텍스트를 만든다.
  3. 학습은 두 단계다. 1단계 지도 미세조정(SFT)에서 양방향 디노이징을 가르치고, 2단계 샘플러 증류(sampler distillation)와 강화학습(RL)을 합친 SD·RL 단계에서 생성 품질을 높이면서 필요한 디노이징 스텝 수를 줄인다.
  4. 엔트로피 기반 샘플러와 적응형 정지(adaptive stopping) 기법으로, 확신도가 높으면 최대 스텝 수를 다 채우지 않고 일찍 멈춰서 지연시간을 최대 4배 줄이면서도 품질 저하가 거의 없다.
  5. Gemma 4와 같은 트랜스포머 구조를 그대로 쓰기 때문에 미세조정 후에도 원래처럼 한 토큰씩 순차 생성(AR)하는 것도 가능해서, 상황에 따라 확산 방식과 AR 방식을 섞어 쓸 여지가 있다.
Figure 1: Pareto plot of quality versus output decoding speed comparing DiffusionGemma to the Gemma 4 model family and other diffusion models. Quality and output speed are calculated as the average across GPQA-Diamond and LiveCodeBench-v6 for all models. Gemma 4 and DiffusionGemma output speeds are measured on a single H100 (FP8, batch size 1).2 Nemotron 14B is measured on a single H100 (bfloat16, batch size 1). LLaDA 2.1 Flash 100B is measured on 8× NVIDIA B200 GPUs (bfloat16, batch size 1).3 Mercury 2 was measured via the public OpenRouter API with high reasoning effort (see Appendix E for speed estimation). The light blue dots and dashed line represent the Gemma 4 family of models without MTP.
Figure 1: Pareto plot of quality versus output decoding speed comparing DiffusionGemma to the Gemma 4 model family and other diffusion models. Quality and output speed are calculated as the average across GPQA-Diamond and LiveCodeBench-v6 for all models. Gemma 4 and DiffusionGemma output speeds are measured on a single H100 (FP8, batch size 1).2 Nemotron 14B is measured on a single H100 (bfloat16, batch size 1). LLaDA 2.1 Flash 100B is measured on 8× NVIDIA B200 GPUs (bfloat16, batch size 1).3 Mercury 2 was measured via the public OpenRouter API with high reasoning effort (see Appendix E for speed estimation). The light blue dots and dashed line represent the Gemma 4 family of models without MTP.
Figure 2: Overview of our two-stage training pipeline that converts an autoregressive model (Gemma 4 26B A4B) into a text diffusion model (DiffusionGemma). Initialized from the AR model weights, the model first undergoes SFT that adapts it to discrete text diffusion and bidirectional attention across 256-token canvases. This is followed by an online sampler distillation and reinforcement learning phase, which jointly maximizes reward-driven generation quality and compresses the denoising steps to unlock ultra-low latency.
Figure 2: Overview of our two-stage training pipeline that converts an autoregressive model (Gemma 4 26B A4B) into a text diffusion model (DiffusionGemma). Initialized from the AR model weights, the model first undergoes SFT that adapts it to discrete text diffusion and bidirectional attention across 256-token canvases. This is followed by an online sampler distillation and reinforcement learning phase, which jointly maximizes reward-driven generation quality and compresses the denoising steps to unlock ultra-low latency.
Table 1: Parameter counts. DiffusionGemma’s architecture is a mixture-of-experts transformer with a vocabulary of 262k tokens. The total number of activated parameters does not include the vision encoder. It includes an additional MLP block for the purpose of self-conditioning.
Total25.2B
Activated3.85B
Vision Encoder550M
Embedder740M
Self-Conditioning7.8M
Active / Total Experts8 / 128
+ 1 shared
Figure 3: Stylized example of discrete diffusion probability paths and parallel sampling trajectory. For illustrative purposes, the state space uses distinct, token-specific vocabularies: adjectives on the horizontal axis and nouns on the vertical axis. As time moves backward from t=1.0 (noise) to t=0.0 (data), the marginal distribution smoothly interpolates, concentrating mass away from the uniform noise distribution and onto valid data modes. Black circles track the discrete jump transitions of an individual sequence realization (from “blue moon” at t=1.0 to “red sunset” at t=0.0), demonstrating how parallel canvas dimensions coordinate non-autoregressively over time.
Figure 3: Stylized example of discrete diffusion probability paths and parallel sampling trajectory. For illustrative purposes, the state space uses distinct, token-specific vocabularies: adjectives on the horizontal axis and nouns on the vertical axis. As time moves backward from t=1.0 (noise) to t=0.0 (data), the marginal distribution smoothly interpolates, concentrating mass away from the uniform noise distribution and onto valid data modes. Black circles track the discrete jump transitions of an individual sequence realization (from “blue moon” at t=1.0 to “red sunset” at t=0.0), demonstrating how parallel canvas dimensions coordinate non-autoregressively over time.
Figure 4: The DiffusionGemma generation pipeline. The process consists of three main stages: 1) Context encoding: The input prompt is processed by the causal encoder to initialize the Key-Value (KV) cache. 2) Denoising loop: A noisy canvas is iteratively refined by the decoder, using bidirectional attention across the canvas and cross-attention to the KV cache, until the text is fully denoised. 3) Encode & append: The finalized clean canvas is passed back through the causal encoder and appended to the KV cache, setting the context for the next block of tokens.
Figure 4: The DiffusionGemma generation pipeline. The process consists of three main stages: 1) Context encoding: The input prompt is processed by the causal encoder to initialize the Key-Value (KV) cache. 2) Denoising loop: A noisy canvas is iteratively refined by the decoder, using bidirectional attention across the canvas and cross-attention to the KV cache, until the text is fully denoised. 3) Encode & append: The finalized clean canvas is passed back through the causal encoder and appended to the KV cache, setting the context for the next block of tokens.
Table 3: Comparison of model performance, TPS and TPF across various benchmarks. TPS excludes prefill time; “−” denotes missing data. For speed measurements: DiffusionGemma and Gemma 4 are measured on 1× H100 (FP8, batch size 1); Nemotron 14B on 1× H100 (bfloat16, batch size 1); LLaDA 2.1 Flash 100B on 8× B200 (bfloat16, batch size 1); Mercury 2 via its public API (see Appendix E for speed estimation). TPS, TPF and total tokens are averaged over the 7 benchmarks for which we have full coverage: AIME 2026, GPQA Diamond, LiveCodeBench-v6, MGSM, HumanEval, LBPP, and Natural2Code. TPS and TPF for Gemma 4 (MTP) are measured using SPEED-Bench (1). The Natural2Code and HiddenMath rows are highlighted as they are proprietary, unleaked evals.
Open-weight ModelsClosed-weight Model
DiffusionGemmaGemma 4LLaDA 2.1 FlashNemotron DiffusionMercury 2
26B A4B26B A4B100B14BUnknown
ModeTDTD (No-think)ARAR (No-think)AR (MTP)AR (MTP, No-think)TD (S Mode)TD (Diffusion Mode)HighMedium
AIME 202669.150.884.257.588.380.080.040.091.782.5
GPQA Diamond73.264.679.867.282.373.768.747.075.266.7
LiveCodeBench-V669.160.671.458.377.172.639.428.679.474.9
Codeforces ELO14299591569105917181529718-19861629
BigBench EH47.640.059.142.264.856.2--48.943.8
GSM8K96.395.896.696.196.796.445.0-96.595.8
MGSM84.880.787.984.392.991.56.869.391.991.2
MMMLU81.576.382.278.086.378.0--81.980.6
MMMU Pro54.366.063.366.773.872.5----
Putnam67.457.174.759.781.072.9-45.873.673.6
HumanEval94.592.798.297.698.897.690.286.098.298.2
BigCodeBench46.041.947.745.950.248.1-33.547.645.3
LBPP81.068.986.374.189.577.345.740.789.285.0
IFEval97.494.597.295.798.797.8-72.197.094.5
Tau2 Retail71.557.575.461.085.579.0----
Tau2 Airline69.049.072.050.076.051.0----
Tau2 Telecom28.132.033.832.043.034.2----
MMLU-Pro77.677.978.879.182.682.6--77.675.5
Natural2Code94.090.196.292.396.394.786.973.379.171.3
HiddenMath80.674.385.477.587.281.6-44.382.782.3
Output Speed (TPS)1479151220420430330337549600547
Tokens Per Forward (TPF)19.7418.761.001.001.401.404.631.79--
Average Total Tokens4,0018295,1841,0257,2071,8164,3719413,8821,222
Figure 5: Adaptive stopping enables DiffusionGemma to dynamically adjust its number of denoising steps to task complexity and domain. We report median, first and third quartiles of the effective denoising steps (Equation 9). See Table 4 for full benchmarks and latency metrics.
Figure 5: Adaptive stopping enables DiffusionGemma to dynamically adjust its number of denoising steps to task complexity and domain. We report median, first and third quartiles of the effective denoising steps (Equation 9). See Table 4 for full benchmarks and latency metrics.
Figure 6: Evolution of downstream performance during SFT. Prior to SFT, the model is incapable of denoising text. Only a moderate amount of SFT is needed to achieve good performance in non-thinking mode, however extended SFT is crucial for the model to learn thinking behaviour. Results use entropy-bounded sampling with adaptive stopping and a maximum of N=192 denoising steps.
Figure 6: Evolution of downstream performance during SFT. Prior to SFT, the model is incapable of denoising text. Only a moderate amount of SFT is needed to achieve good performance in non-thinking mode, however extended SFT is crucial for the model to learn thinking behaviour. Results use entropy-bounded sampling with adaptive stopping and a maximum of N=192 denoising steps.
Table 4: Performance and latency metrics of DiffusionGemma TD in Thinking vs. No-Thinking mode. We report accuracy/score, Tokens Per Forward (TPF, higher is faster), Tokens Per Second (TPS), Effective Denoising Steps (DNS, lower is faster), Total Forwards, Total Tokens, and End-to-End Latency per sample in seconds (excluding prefill time). The latency metrics are averaged across all samples within each benchmark.
Score (↑)TPF (↑)TPS (↑)Effective DNS (↓)Total Forwards (↓)Total TokensE2E Time (s) (↓)
BenchmarkThinkNo-ThinkThinkNo-ThinkThinkNo-ThinkThinkNo-ThinkThinkNo-ThinkThinkNo-ThinkThinkNo-Think
AIME 202669.150.819.316.71365.41333.012.614.1390.691.16,4451,3094.720.98
GPQA Diamond73.264.616.716.51207.81330.215.113.4443.448.15,6477264.680.55
LiveCodeBench-V669.160.618.516.91278.31333.413.814.0581.8195.77,5341,8475.891.39
Codeforces ELO142995915.114.0950.51040.317.118.5959.6521.111,6224,27912.234.11
BigBench EH47.640.020.817.71390.21415.111.912.8434.968.19,0621,2336.520.87
GSM8K96.395.823.224.11866.21966.49.17.543.413.28832980.470.15
MGSM84.880.719.016.81526.71367.511.511.963.619.51,0852910.710.21
MMMU Pro54.366.017.715.41351.31255.113.815.3191.631.63,1784722.350.38
Putnam67.457.118.116.01330.81282.413.414.4303.877.44,7251,1033.550.86
HumanEval94.592.723.024.31838.21981.29.48.055.614.31,1743050.640.15
BigCodeBench46.041.919.619.41560.11579.011.310.177.321.81,4103940.900.25
LBPP81.068.920.519.21509.81545.411.610.9264.379.44,7308593.130.56
IFEval97.494.517.29.01368.3732.113.014.4100.824.01,4642391.070.33
Natural2Code94.090.121.121.01682.71706.510.59.670.524.71,3914650.830.27
HiddenMath80.674.321.019.41591.21564.711.311.5206.449.73,5018442.200.54
Figure 7: Downstream performance during SFT improves log-linearly with training progress. The log-linear trend for thinking performance starts off at a lower point yet exhibits a steeper slope than non-thinking mode. Results use the EntropyBounded sampler with adaptive stopping and a maximum of N=192 denoising steps.
Figure 7: Downstream performance during SFT improves log-linearly with training progress. The log-linear trend for thinking performance starts off at a lower point yet exhibits a steeper slope than non-thinking mode. Results use the EntropyBounded sampler with adaptive stopping and a maximum of N=192 denoising steps.
Figure 8: SD⋅RL training simultaneously increases average reward and reduces the effective denoising steps of the online teacher (training metrics shown in buckets of 200 steps). Both effects together push the quality-speed Pareto frontier of the model.
Figure 8: SD⋅RL training simultaneously increases average reward and reduces the effective denoising steps of the online teacher (training metrics shown in buckets of 200 steps). Both effects together push the quality-speed Pareto frontier of the model.
Table 6: PubMedQA performance of the finetuned model with LoRA rank 4. Finetuning leads to a slight increase in accuracy on a model with a good base performance.
ModelEffective Denoising StepsAccuracy (%)BLEU
DiffusionGemma18.0975.610.76
+ LoRA finetuning31.5776.6220.67
Figure 9: SD⋅RL significantly advances the quality-speed Pareto frontier. The SD⋅RL configuration uses the DiffusionGemma sampler with a maximum of N=48 denoising steps. The SFT frontier is derived by sweeping N from 48 to 192 (note that higher N yields milder temperature annealing). Both quality (y-axis) and inference efficiency (x-axes, see Section 3.4) are calculated as the average between GPQA-Diamond and LiveCodeBench-v6 in thinking mode, averaged over 3 seeds.
Figure 9: SD⋅RL significantly advances the quality-speed Pareto frontier. The SD⋅RL configuration uses the DiffusionGemma sampler with a maximum of N=48 denoising steps. The SFT frontier is derived by sweeping N from 48 to 192 (note that higher N yields milder temperature annealing). Both quality (y-axis) and inference efficiency (x-axes, see Section 3.4) are calculated as the average between GPQA-Diamond and LiveCodeBench-v6 in thinking mode, averaged over 3 seeds.
Figure 10: Performance vs. number of denoising steps N, without adaptive stopping or temperature annealing. Performance is measured as the average score over GPQA-Diamond, LiveCodeBench-v6 (3 seeds). To remove confounding effects of early stopping and temperature annealing, we disabled adaptive stopping and use temperature τt=1.
Figure 10: Performance vs. number of denoising steps N, without adaptive stopping or temperature annealing. Performance is measured as the average score over GPQA-Diamond, LiveCodeBench-v6 (3 seeds). To remove confounding effects of early stopping and temperature annealing, we disabled adaptive stopping and use temperature τt=1.
Table 7: Summary of downstream finetuning hyperparameters for the Sudoku and PubMedQA recipes. Both LoRA recipes use adapters applied to all linear layers and train for 2,000 steps. A full-weight alternative is also provided for Sudoku, using Adafactor to manage memory on 8 GPUs.
HyperparameterSudoku (LoRA)Sudoku (Full)PubMedQA
LoRA rank8—4
Canvas size256256128
Number of canvases112
Prompt length2562561024
Batch size282
Peak learning rate3×10−41.125×10−41.0×10−4
End learning rate3×10−51.125×10−51.0×10−5
Training steps8,0002,0002,000
OptimizerAdamAdafactorAdam
LR scheduleCosine with warmupCosine with warmupCosine with warmup
Warmup iterations400100100
Weight decay10−410−410−4
Min. hardware2× A100 80GB8× A100 80GB2× A100 80GB
Figure 11: Per-step GPU time breakdown: DiffusionGemma processes 256 tokens per step with only a 3.2× increase in per-step latency compared to single-token AR generation. Serving a single request (batch size 1) on a H100, FP8 precision, with 4096 input tokens, 1024 output tokens.
Figure 11: Per-step GPU time breakdown: DiffusionGemma processes 256 tokens per step with only a 3.2× increase in per-step latency compared to single-token AR generation. Serving a single request (batch size 1) on a H100, FP8 precision, with 4096 input tokens, 1024 output tokens.
Figure 12: Trade-off between total and per-user throughput of the Gemma 4 AR model (with and without MTP) and DiffusionGemma. In the low batch size regime, DiffusionGemma offers substantially higher TPS per user and higher total throughput. It is only at moderate batch sizes (around 32 concurrent requests) that AR models begin to have a throughput advantage. All models are run on an H100 with FP8 precision using the PG-19 benchmark (4096 input tokens, 1024 output tokens); the Gemma 4 AR (MTP) model uses a draft length of 4.
Figure 12: Trade-off between total and per-user throughput of the Gemma 4 AR model (with and without MTP) and DiffusionGemma. In the low batch size regime, DiffusionGemma offers substantially higher TPS per user and higher total throughput. It is only at moderate batch sizes (around 32 concurrent requests) that AR models begin to have a throughput advantage. All models are run on an H100 with FP8 precision using the PG-19 benchmark (4096 input tokens, 1024 output tokens); the Gemma 4 AR (MTP) model uses a draft length of 4.

실제로 확인된 결과

  • 평가 전반 평균으로 순전파 1회당 약 20개 토큰을 생성하며, 단일 NVIDIA H100 GPU에서 초당 약 1,500개 토큰(TPS)을 낸다.
  • 이는 최신 추측 디코딩(speculative decoding)을 적용한 AR 모델(순전파당 약 3~6토큰)보다 훨씬 빠른 수준이다.
  • 적응형 정지를 적용하면 최대 48스텝 예산 중 평균 약 12스텝만 써서, 품질 저하 없이 지연시간을 약 4배 줄인다.
  • 256토큰을 한 스텝에 처리하면서도, 단일 토큰 AR 생성 대비 스텝당 지연시간 증가는 3.2배에 그친다(H100, FP8, 배치 크기 1 기준).
  • AR 모드로 되돌려 생성해도 성능 저하가 크지 않아, 같은 가중치로 AR 생성도 가능하다.

어디에 쓸 수 있나

  • 응답 속도가 중요한 저지연 챗봇이나 코드 어시스턴트 서빙
  • 적은 동시 요청(낮은 배치 크기) 환경에서 GPU 활용도를 높이는 추론 서버 구성
  • 라이선스가 개방(Apache 2.0)되어 있어 특정 도메인(예: 의료 QA, 음성 인식)에 맞춘 경량 미세조정
  • 지연시간 제약과 작업 난이도에 따라 확산 생성과 AR 생성을 동적으로 나눠 쓰는 하이브리드 서빙

한계와 남은 검증

  • 동시 요청(배치 크기)이 약 32건 이상으로 늘어나면 AR 모델이 처리량 면에서 다시 유리해진다.
  • 확산 방식으로 미세조정하면서 원래 AR 기준 대비 약간의 성능 저하가 발생한다.
  • Mercury 2 등 폐쇄형 API 모델의 속도는 직접 측정이 아니라 OpenRouter API 응답을 통한 추정치이며 오차 요인이 있다.
  • 논문 본문 중간 상세 실험 섹션 및 부록 일부(F, G.3 등)는 이번 자료에서 생략되어 세부 수치를 모두 확인하지 못했다.
  • 이산 확산 특성상 각 위치 갱신이 다른 위치의 동시 결정을 보지 못해 국소적인 문법 불일치가 생길 수 있다.

왜 중요한가

챗봇이나 코딩 도우미처럼 응답 속도가 중요한 서비스에서 GPU 한 대로도 훨씬 빠른 텍스트 생성이 가능해질 수 있다는 뜻이다. 또한 가중치를 공개했기 때문에 연구자와 개발자가 직접 이산 확산 모델의 작동 방식을 들여다보고 자신의 용도에 맞게 손쉽게 재조정할 수 있다.

이 논문의 용어

  • 이산 확산(discrete diffusion) · 연속적인 이미지 확산과 달리, 어휘 안의 실제 토큰 상태들 사이에서 노이즈를 섞고 되돌리는 방식의 생성 모델
  • 자기회귀(AR) 모델 · 왼쪽에서 오른쪽으로 토큰을 하나씩 순서대로 생성하는 기존 언어모델 방식
  • 혼합전문가(MoE) · 여러 개의 전문 서브네트워크 중 일부만 골라 활성화해 계산량을 줄이는 모델 구조
  • 적응형 정지(adaptive stopping) · 모델의 예측 확신도가 충분히 높아지면 최대 스텝 수를 다 채우지 않고 디노이징을 일찍 끝내는 기법
  • 샘플러 증류(sampler distillation) · 고품질이지만 느린 생성 절차의 결과를, 더 적은 스텝으로도 비슷하게 낼 수 있도록 모델에 가르치는 학습 기법

저자 · DiffusionGemma Team

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사

그림 출처: DiffusionGemma Team et al., arXiv:2608.00146, CC BY 4.0