DiffusionGemma Technical Report

arXiv:2608.001462026-07-30

Google's DiffusionGemma is an experimental open-weight model that refines 256-token blocks in parallel to generate text far faster than conventional token-by-token AR models

Instead of generating one token at a time like standard autoregressive (AR) models, DiffusionGemma iteratively refines blocks of 256 tokens at once using discrete diffusion. It was created not by training from scratch but by fine-tuning Gemma 4, a mixture-of-experts (MoE) model with 3.8B activated and 25.2B total parameters, using less than 10% of the original AR model's training token budget. Averaged across its evaluation suite, it produces about 20 tokens per forward pass and roughly 1,500 output tokens per second on a single H100 GPU, substantially faster than AR models even with state-of-the-art speculative decoding.

METAL MEDIA explanatory visual

How DiffusionGemma's two-stage pipeline produces fast text

Evidence statusMeasured results reported

Starting point: Gemma 4 AR modelInitialized from the pretrained Gemma 4 26B A4B MoE checkpoint (3.8B activated, 25.2B total parameters) rather than training from scratch
Stage 1: SFTSupervised fine-tuning teaches bidirectional denoising over 256-token canvases, using under 10% of the AR model's original training token budget
Stage 2: SD·RLSampler distillation combined with reinforcement learning jointly boosts reward-driven quality and compresses the number of denoising steps needed
Inference: block-wise diffusion decodingStarting from a random noise canvas, an entropy-bounded sampler with adaptive stopping finishes a 256-token block in about 12 steps on average, then appends it to the KV cache
Result: new speed-quality Pareto frontierAbout 1,500 tokens per second and 20 tokens per forward pass on a single H100, beating AR models even with state-of-the-art speculative decoding

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Conventional AR models generate one token at a time, which leaves GPU compute underutilized at low request volumes because time is dominated by moving weights and KV cache from memory (a memory bandwidth bottleneck).
DiffusionGemma sidesteps this with discrete diffusion: it starts from a 256-token canvas of random noise and iteratively refines the whole block in parallel using bidirectional attention.
Training has two stages: supervised fine-tuning (SFT) first teaches the model bidirectional denoising, then a second stage called SD·RL combines sampler distillation with reinforcement learning to jointly improve quality and cut the number of denoising steps needed.
An entropy-bounded sampler with adaptive stopping lets the model halt denoising early once its predictions are confident and stable, cutting latency by up to roughly 4x without sacrificing quality.
Because it shares the exact same transformer architecture as Gemma 4, the fine-tuned weights can still be run in standard autoregressive mode, leaving room for hybrid diffusion-AR decoding depending on the situation.

Figure 1: Pareto plot of quality versus output decoding speed comparing DiffusionGemma to the Gemma 4 model family and other diffusion models. Quality and output speed are calculated as the average across GPQA-Diamond and LiveCodeBench-v6 for all models. Gemma 4 and DiffusionGemma output speeds are measured on a single H100 (FP8, batch size 1).2 Nemotron 14B is measured on a single H100 (bfloat16, batch size 1). LLaDA 2.1 Flash 100B is measured on 8× NVIDIA B200 GPUs (bfloat16, batch size 1).3 Mercury 2 was measured via the public OpenRouter API with high reasoning effort (see Appendix E for speed estimation). The light blue dots and dashed line represent the Gemma 4 family of models without MTP.

Figure 2: Overview of our two-stage training pipeline that converts an autoregressive model (Gemma 4 26B A4B) into a text diffusion model (DiffusionGemma). Initialized from the AR model weights, the model first undergoes SFT that adapts it to discrete text diffusion and bidirectional attention across 256-token canvases. This is followed by an online sampler distillation and reinforcement learning phase, which jointly maximizes reward-driven generation quality and compresses the denoising steps to unlock ultra-low latency.

Table 1: Parameter counts. DiffusionGemma’s architecture is a mixture-of-experts transformer with a vocabulary of 262k tokens. The total number of activated parameters does not include the vision encoder. It includes an additional MLP block for the purpose of self-conditioning.
Total	25.2B
Activated	3.85B
Vision Encoder	550M
Embedder	740M
Self-Conditioning	7.8M
Active / Total Experts	8 / 128
	+ 1 shared

Figure 3: Stylized example of discrete diffusion probability paths and parallel sampling trajectory. For illustrative purposes, the state space uses distinct, token-specific vocabularies: adjectives on the horizontal axis and nouns on the vertical axis. As time moves backward from t=1.0 (noise) to t=0.0 (data), the marginal distribution smoothly interpolates, concentrating mass away from the uniform noise distribution and onto valid data modes. Black circles track the discrete jump transitions of an individual sequence realization (from “blue moon” at t=1.0 to “red sunset” at t=0.0), demonstrating how parallel canvas dimensions coordinate non-autoregressively over time.

Figure 4: The DiffusionGemma generation pipeline. The process consists of three main stages: 1) Context encoding: The input prompt is processed by the causal encoder to initialize the Key-Value (KV) cache. 2) Denoising loop: A noisy canvas is iteratively refined by the decoder, using bidirectional attention across the canvas and cross-attention to the KV cache, until the text is fully denoised. 3) Encode & append: The finalized clean canvas is passed back through the causal encoder and appended to the KV cache, setting the context for the next block of tokens.

Table 3: Comparison of model performance, TPS and TPF across various benchmarks. TPS excludes prefill time; “−” denotes missing data. For speed measurements: DiffusionGemma and Gemma 4 are measured on 1× H100 (FP8, batch size 1); Nemotron 14B on 1× H100 (bfloat16, batch size 1); LLaDA 2.1 Flash 100B on 8× B200 (bfloat16, batch size 1); Mercury 2 via its public API (see Appendix E for speed estimation). TPS, TPF and total tokens are averaged over the 7 benchmarks for which we have full coverage: AIME 2026, GPQA Diamond, LiveCodeBench-v6, MGSM, HumanEval, LBPP, and Natural2Code. TPS and TPF for Gemma 4 (MTP) are measured using SPEED-Bench (1). The Natural2Code and HiddenMath rows are highlighted as they are proprietary, unleaked evals.
	Open-weight Models	Closed-weight Model
	DiffusionGemma	Gemma 4	LLaDA 2.1 Flash	Nemotron Diffusion	Mercury 2
	26B A4B	26B A4B	100B	14B	Unknown
Mode	TD	TD (No-think)	AR	AR (No-think)	AR (MTP)	AR (MTP, No-think)	TD (S Mode)	TD (Diffusion Mode)	High	Medium
AIME 2026	69.1	50.8	84.2	57.5	88.3	80.0	80.0	40.0	91.7	82.5
GPQA Diamond	73.2	64.6	79.8	67.2	82.3	73.7	68.7	47.0	75.2	66.7
LiveCodeBench-V6	69.1	60.6	71.4	58.3	77.1	72.6	39.4	28.6	79.4	74.9
Codeforces ELO	1429	959	1569	1059	1718	1529	718	-	1986	1629
BigBench EH	47.6	40.0	59.1	42.2	64.8	56.2	-	-	48.9	43.8
GSM8K	96.3	95.8	96.6	96.1	96.7	96.4	45.0	-	96.5	95.8
MGSM	84.8	80.7	87.9	84.3	92.9	91.5	6.8	69.3	91.9	91.2
MMMLU	81.5	76.3	82.2	78.0	86.3	78.0	-	-	81.9	80.6
MMMU Pro	54.3	66.0	63.3	66.7	73.8	72.5	-	-	-	-
Putnam	67.4	57.1	74.7	59.7	81.0	72.9	-	45.8	73.6	73.6
HumanEval	94.5	92.7	98.2	97.6	98.8	97.6	90.2	86.0	98.2	98.2
BigCodeBench	46.0	41.9	47.7	45.9	50.2	48.1	-	33.5	47.6	45.3
LBPP	81.0	68.9	86.3	74.1	89.5	77.3	45.7	40.7	89.2	85.0
IFEval	97.4	94.5	97.2	95.7	98.7	97.8	-	72.1	97.0	94.5
Tau2 Retail	71.5	57.5	75.4	61.0	85.5	79.0	-	-	-	-
Tau2 Airline	69.0	49.0	72.0	50.0	76.0	51.0	-	-	-	-
Tau2 Telecom	28.1	32.0	33.8	32.0	43.0	34.2	-	-	-	-
MMLU-Pro	77.6	77.9	78.8	79.1	82.6	82.6	-	-	77.6	75.5
Natural2Code	94.0	90.1	96.2	92.3	96.3	94.7	86.9	73.3	79.1	71.3
HiddenMath	80.6	74.3	85.4	77.5	87.2	81.6	-	44.3	82.7	82.3
Output Speed (TPS)	1479	1512	204	204	303	303	375	49	600	547
Tokens Per Forward (TPF)	19.74	18.76	1.00	1.00	1.40	1.40	4.63	1.79	-	-
Average Total Tokens	4,001	829	5,184	1,025	7,207	1,816	4,371	941	3,882	1,222

Figure 5: Adaptive stopping enables DiffusionGemma to dynamically adjust its number of denoising steps to task complexity and domain. We report median, first and third quartiles of the effective denoising steps (Equation 9). See Table 4 for full benchmarks and latency metrics.

Figure 6: Evolution of downstream performance during SFT. Prior to SFT, the model is incapable of denoising text. Only a moderate amount of SFT is needed to achieve good performance in non-thinking mode, however extended SFT is crucial for the model to learn thinking behaviour. Results use entropy-bounded sampling with adaptive stopping and a maximum of N=192 denoising steps.

Table 4: Performance and latency metrics of DiffusionGemma TD in Thinking vs. No-Thinking mode. We report accuracy/score, Tokens Per Forward (TPF, higher is faster), Tokens Per Second (TPS), Effective Denoising Steps (DNS, lower is faster), Total Forwards, Total Tokens, and End-to-End Latency per sample in seconds (excluding prefill time). The latency metrics are averaged across all samples within each benchmark.
	Score (↑)	TPF (↑)	TPS (↑)	Effective DNS (↓)	Total Forwards (↓)	Total Tokens	E2E Time (s) (↓)
Benchmark	Think	No-Think	Think	No-Think	Think	No-Think	Think	No-Think	Think	No-Think	Think	No-Think	Think	No-Think
AIME 2026	69.1	50.8	19.3	16.7	1365.4	1333.0	12.6	14.1	390.6	91.1	6,445	1,309	4.72	0.98
GPQA Diamond	73.2	64.6	16.7	16.5	1207.8	1330.2	15.1	13.4	443.4	48.1	5,647	726	4.68	0.55
LiveCodeBench-V6	69.1	60.6	18.5	16.9	1278.3	1333.4	13.8	14.0	581.8	195.7	7,534	1,847	5.89	1.39
Codeforces ELO	1429	959	15.1	14.0	950.5	1040.3	17.1	18.5	959.6	521.1	11,622	4,279	12.23	4.11
BigBench EH	47.6	40.0	20.8	17.7	1390.2	1415.1	11.9	12.8	434.9	68.1	9,062	1,233	6.52	0.87
GSM8K	96.3	95.8	23.2	24.1	1866.2	1966.4	9.1	7.5	43.4	13.2	883	298	0.47	0.15
MGSM	84.8	80.7	19.0	16.8	1526.7	1367.5	11.5	11.9	63.6	19.5	1,085	291	0.71	0.21
MMMU Pro	54.3	66.0	17.7	15.4	1351.3	1255.1	13.8	15.3	191.6	31.6	3,178	472	2.35	0.38
Putnam	67.4	57.1	18.1	16.0	1330.8	1282.4	13.4	14.4	303.8	77.4	4,725	1,103	3.55	0.86
HumanEval	94.5	92.7	23.0	24.3	1838.2	1981.2	9.4	8.0	55.6	14.3	1,174	305	0.64	0.15
BigCodeBench	46.0	41.9	19.6	19.4	1560.1	1579.0	11.3	10.1	77.3	21.8	1,410	394	0.90	0.25
LBPP	81.0	68.9	20.5	19.2	1509.8	1545.4	11.6	10.9	264.3	79.4	4,730	859	3.13	0.56
IFEval	97.4	94.5	17.2	9.0	1368.3	732.1	13.0	14.4	100.8	24.0	1,464	239	1.07	0.33
Natural2Code	94.0	90.1	21.1	21.0	1682.7	1706.5	10.5	9.6	70.5	24.7	1,391	465	0.83	0.27
HiddenMath	80.6	74.3	21.0	19.4	1591.2	1564.7	11.3	11.5	206.4	49.7	3,501	844	2.20	0.54

Figure 7: Downstream performance during SFT improves log-linearly with training progress. The log-linear trend for thinking performance starts off at a lower point yet exhibits a steeper slope than non-thinking mode. Results use the EntropyBounded sampler with adaptive stopping and a maximum of N=192 denoising steps.

Figure 8: SD⋅RL training simultaneously increases average reward and reduces the effective denoising steps of the online teacher (training metrics shown in buckets of 200 steps). Both effects together push the quality-speed Pareto frontier of the model.

Table 6: PubMedQA performance of the finetuned model with LoRA rank 4. Finetuning leads to a slight increase in accuracy on a model with a good base performance.
Model	Effective Denoising Steps	Accuracy (%)	BLEU
DiffusionGemma	18.09	75.6	10.76
+ LoRA finetuning	31.57	76.62	20.67

Figure 9: SD⋅RL significantly advances the quality-speed Pareto frontier. The SD⋅RL configuration uses the DiffusionGemma sampler with a maximum of N=48 denoising steps. The SFT frontier is derived by sweeping N from 48 to 192 (note that higher N yields milder temperature annealing). Both quality (y-axis) and inference efficiency (x-axes, see Section 3.4) are calculated as the average between GPQA-Diamond and LiveCodeBench-v6 in thinking mode, averaged over 3 seeds.

Figure 10: Performance vs. number of denoising steps N, without adaptive stopping or temperature annealing. Performance is measured as the average score over GPQA-Diamond, LiveCodeBench-v6 (3 seeds). To remove confounding effects of early stopping and temperature annealing, we disabled adaptive stopping and use temperature τt=1.

Table 7: Summary of downstream finetuning hyperparameters for the Sudoku and PubMedQA recipes. Both LoRA recipes use adapters applied to all linear layers and train for 2,000 steps. A full-weight alternative is also provided for Sudoku, using Adafactor to manage memory on 8 GPUs.
Hyperparameter	Sudoku (LoRA)	Sudoku (Full)	PubMedQA
LoRA rank	8	—	4
Canvas size	256	256	128
Number of canvases	1	1	2
Prompt length	256	256	1024
Batch size	2	8	2
Peak learning rate	3×10−4	1.125×10−4	1.0×10−4
End learning rate	3×10−5	1.125×10−5	1.0×10−5
Training steps	8,000	2,000	2,000
Optimizer	Adam	Adafactor	Adam
LR schedule	Cosine with warmup	Cosine with warmup	Cosine with warmup
Warmup iterations	400	100	100
Weight decay	10−4	10−4	10−4
Min. hardware	2× A100 80GB	8× A100 80GB	2× A100 80GB

Figure 11: Per-step GPU time breakdown: DiffusionGemma processes 256 tokens per step with only a 3.2× increase in per-step latency compared to single-token AR generation. Serving a single request (batch size 1) on a H100, FP8 precision, with 4096 input tokens, 1024 output tokens.

Figure 12: Trade-off between total and per-user throughput of the Gemma 4 AR model (with and without MTP) and DiffusionGemma. In the low batch size regime, DiffusionGemma offers substantially higher TPS per user and higher total throughput. It is only at moderate batch sizes (around 32 concurrent requests) that AR models begin to have a throughput advantage. All models are run on an H100 with FP8 precision using the PG-19 benchmark (4096 input tokens, 1024 output tokens); the Gemma 4 AR (MTP) model uses a draft length of 4.

Findings

Averaged across the full evaluation suite, DiffusionGemma generates about 20 tokens per forward pass and achieves roughly 1,500 output tokens per second (TPS) on a single NVIDIA H100 GPU.
This is substantially faster than AR models using state-of-the-art speculative decoding, which typically achieve only about 3-6 tokens per forward pass.
With adaptive stopping, the model averages about 12 effective denoising steps out of a maximum budget of 48, yielding roughly a 4x reduction in latency without sacrificing generation quality.
Although it processes 256 tokens per step, per-step latency only increases 3.2x compared to single-token AR generation (measured on H100, FP8, batch size 1).
The fine-tuned weights can still be run for standard AR generation with only minor performance degradation.

Where it can be used

Serving latency-critical chatbots or coding assistants where per-user response speed matters
Inference setups with low concurrency (small batch sizes) where GPU compute would otherwise be underutilized
Lightweight domain-specific fine-tuning (e.g., medical QA, speech recognition) enabled by the permissive Apache 2.0 license
Hybrid serving that dynamically routes requests between diffusion and AR decoding depending on latency constraints and task complexity

Limits and open work

At moderate batch sizes (around 32 concurrent requests), AR models regain a throughput advantage over DiffusionGemma.
Diffusion fine-tuning introduces some performance degradation compared to the original AR baseline.
Speed figures for closed-source competitors like Mercury 2 are estimated indirectly via the OpenRouter API rather than measured directly, introducing potential error.
Large portions of the paper's middle sections and some appendices (e.g., Appendix F, G.3) were omitted from the provided material, so not all detailed results could be reviewed.
Because discrete diffusion updates each position without seeing simultaneous choices at other positions, it can occasionally produce local grammatical inconsistencies.

Why it matters

For latency-sensitive services like chatbots or coding assistants, this suggests a single GPU could deliver dramatically faster text generation than current AR serving setups. The open-weight release under Apache 2.0 also means researchers and developers can inspect the mechanics of discrete diffusion directly and adapt the model to their own domains with relatively low compute.

Terms in this paper

discrete diffusion · A generative approach that adds and removes noise directly among actual vocabulary tokens, unlike continuous-space diffusion used for images
autoregressive (AR) model · The standard language model approach that generates tokens one by one, left to right
mixture-of-experts (MoE) · A model architecture that activates only a subset of specialized sub-networks per input to reduce compute
adaptive stopping · A technique that ends the denoising process early once the model's predictions become confident and stable, instead of always running the maximum number of steps
sampler distillation · A training technique that teaches the model to reproduce high-quality but slow generation outcomes using fewer denoising steps

Original abstract (English)

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Authors · DiffusionGemma Team

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: DiffusionGemma Team et al., arXiv:2608.00146, CC BY 4.0