Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack

arXiv:2608.181822026-08-20

Intel engineers made BERT-style models run up to 5.8x faster on server CPUs, using only PyTorch's built-in tools

BERT-family models like BERT and DistilBERT are still heavily used in industry for tasks like classification and search, even though everyone talks about large language models now. This work plugs SmoothQuant, a technique that squeezes model numbers into 8-bit integers (INT8) without losing much accuracy, directly into PyTorch's own compiler stack instead of requiring separate third-party tools. The result is up to 5.8x faster processing on Intel Xeon server chips with barely any accuracy loss.

METAL MEDIA explanatory visual

Intel engineers made BERT-style models run up to 5.8x faster on server CPUs, using only PyTorch's built-in tools

01Integrated SmoothQuant (a method that shifts numerical 'difficulty' from activations to weights before compressing to INT8) into TorchAO, PyTorch's native quantization library, so users don't need external tools.
02Added fusion passes in TorchInductor (PyTorch's compiler backend) to merge INT8 matrix-multiplication operations with scaling and bias steps into a single efficient operator, avoiding wasted time converting data layouts at runtime.
03Built custom low-level kernels using AVX512_VNNI and AMX (special CPU instruction sets for fast integer math) so both older Ice Lake chips and newer Granite Rapids chips get the best available speed.
04Devised a workaround (s8s8-to-u8s8 transformation) so INT8 quantization still works efficiently on Ice Lake CPUs that lack certain instruction support.
05Benchmarked on BERT-large, DistilBERT, and XLM-RoBERTa: throughput improved 1.9x-2.6x on Ice Lake and 4.2x-5.8x on Granite Rapids versus FP32 (32-bit floating point, the default precision), with less than 1% accuracy loss on SQuAD and MultiNLI tasks.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Integrated SmoothQuant (a method that shifts numerical 'difficulty' from activations to weights before compressing to INT8) into TorchAO, PyTorch's native quantization library, so users don't need external tools.
Added fusion passes in TorchInductor (PyTorch's compiler backend) to merge INT8 matrix-multiplication operations with scaling and bias steps into a single efficient operator, avoiding wasted time converting data layouts at runtime.
Built custom low-level kernels using AVX512_VNNI and AMX (special CPU instruction sets for fast integer math) so both older Ice Lake chips and newer Granite Rapids chips get the best available speed.
Devised a workaround (s8s8-to-u8s8 transformation) so INT8 quantization still works efficiently on Ice Lake CPUs that lack certain instruction support.
Benchmarked on BERT-large, DistilBERT, and XLM-RoBERTa: throughput improved 1.9x-2.6x on Ice Lake and 4.2x-5.8x on Granite Rapids versus FP32 (32-bit floating point, the default precision), with less than 1% accuracy loss on SQuAD and MultiNLI tasks.

Figure 1: SmoothQuant workflow built on TorchAO and TorchInductor. TorchAO prepares and quantizes the model, TorchInductor fuses INT8 subgraphs, and the CPP backend selects between oneDNN and template-based GEMM kernels. C++ code is generated and compiled for deployment.

Table 1: Hardware platforms used for baseline collection.
Xeon Generation	Codename	Model	Physical Cores	Instruction Support
3rd Gen	Ice Lake	8358	64	AVX512F, AVX512_VNNI
6th Gen	Granite Rapids	6980P	256	AVX512F, AVX512_VNNI, AMX

Table 2: Baseline collection setup.
Item	Value
Model	BERT-large
Input shape	Batch size 1, sequence length 256
Precision modes	FP32 on both platforms; BF16 on Granite Rapids
Launch strategy	Multi-instance with weight-sharing on each socket
Cores per instance	4
Number of instances	Number of physical cores / 4
Deployment path	AOTI enabled
Input data	Synthetic

Table 3: Baseline hotspot breakdown for the BERT-large analysis case under the same multi-instance setup used for throughput benchmarking. Operator shares are aggregated from representative profiler logs. The linear-block consists of GEMM and fused post-operations such as bias addition, activation, and residual addition.
Baseline	Linear-Block	Attention	LayerNorm	Other
FP32 on Ice Lake	88.5%	10.4%	0.6%	0.5%
FP32 on Granite Rapids	86.9%	11.8%	0.8%	0.5%
BF16 on Granite Rapids	72.70%	19.75%	5.99%	1.56%

Table 4: Estimated peak compute and measured memory bandwidth per socket, balance point for different data types on the evaluated platforms.
Platform	FP32 Throughput (TFLOPS)	BF16 Throughput (TFLOPS)	INT8 Throughput (TOPS)	Memory Bandwidth (GB/s)	FP32 Balance Point (FLOPs/byte)	BF16 Balance Point (FLOPs/byte)	INT8 Balance Point (OPs/byte)
Ice Lake	6.76	N/A	27.03	175.00	38.63	N/A	154.46
Granite Rapids	19.66	262.14	524.29	700.00	28.09	374.49	748.99

Table 5: BERT-large GEMM shapes and their arithmetic intensity.
		Arithmetic Intensity (ops/byte)
GEMM Shape M × K × N	Occurrences per forward pass	FP32	BF16	INT8
256×1024×1024	96	85.33	146.29	227.56
256×1024×4096	24	97.52	163.84	248.24
256×4096×1024	24	97.52	186.18	341.33

Table 6: Effective memory bandwidth and balance point for INT8 GEMM on Granite Rapids.
GEMM Shape	Effective Memory Bandwidth (GB/s)	Arithmetic Intensity (OPs/byte)	Balance Point (OPs/byte)
256×1024×1024	807.69	227.56	649.12
256×1024×4096	819.15	248.24	640.04
256×4096×1024	875.00	341.33	599.19

Table 7: Estimated GEMM throughput and speedup on Granite Rapids.
GEMM Shape	FP32 Throughput (TFLOPS)	BF16 Throughput (TFLOPS)	INT8 Throughput (TOPS)	INT8 vs FP32 Speedup	INT8 vs BF16 Speedup
256×1024×1024	19.66	102.40	183.79	9.35×	1.79×
256×1024×4096	19.66	114.69	203.35	10.34×	1.77×
256×4096×1024	19.66	130.33	298.67	15.19×	2.29×
Weighted Average	19.66	114.69	218.87	11.13×	1.91×

Table 8: Models used for the performance benchmark.
Model	Model ID
BERT-large	bert-large-uncased
DistilBERT	distilbert-base-uncased
XLM-RoBERTa	xlm-roberta-base

Table 9: Task-finetuned checkpoints used for accuracy evaluation. The model marked by * is finetuned on SQuAD 2.0 but still workable for the test on SQuAD 1.1.
Dataset	Model	Model ID
SQuAD 1.1	BERT-large	google-bert/bert-large-uncased-whole-word-masking-finetuned-squad
SQuAD 1.1	DistilBERT	distilbert/distilbert-base-cased-distilled-squad
SQuAD 1.1	XLM-RoBERTa	deepset/xlm-roberta-base-squad2*
MultiNLI	BERT-large	yoshitomo-matsubara/bert-large-uncased-mnli
MultiNLI	DistilBERT	typeform/distilbert-base-uncased-mnli
MultiNLI	XLM-RoBERTa	symanto/xlm-roberta-base-snli-mnli-anli-xnli

Table 10: Quantization methods used in the experiments.
Method	Short Name
No quantization, no AMP	FP32
No quantization, with AMP	BF16
SmoothQuant, static quantization	Smooth-Static
SmoothQuant, dynamic quantization	Smooth-Dynamic

Table 11: INT8 speedup over FP32 on Ice Lake.
Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	2.22×	3.59×
BERT-large	Smooth-Static	2.58×	3.87×
DistilBERT	Smooth-Dynamic	1.90×	3.31×
DistilBERT	Smooth-Static	2.30×	3.35×
XLM-RoBERTa	Smooth-Dynamic	1.86×	3.05×
XLM-RoBERTa	Smooth-Static	2.17×	3.71×

Table 12: INT8 speedup over FP32 on Granite Rapids.
Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	5.02×	9.92×
BERT-large	Smooth-Static	5.82×	8.67×
DistilBERT	Smooth-Dynamic	4.24×	7.60×
DistilBERT	Smooth-Static	5.37×	8.84×
XLM-RoBERTa	Smooth-Dynamic	4.17×	7.58×
XLM-RoBERTa	Smooth-Static	5.41×	8.71×

Table 13: INT8 speedup over BF16 on Granite Rapids.
Model	Method	Throughput Speedup	Linear-Block Speedup
BERT-large	Smooth-Dynamic	1.37×	1.84×
BERT-large	Smooth-Static	1.59×	1.61×
DistilBERT	Smooth-Dynamic	0.98×	1.45×
DistilBERT	Smooth-Static	1.24×	1.69×
XLM-RoBERTa	Smooth-Dynamic	0.99×	1.50×
XLM-RoBERTa	Smooth-Static	1.29×	1.67×

Table 14: Realized speedup of BERT-large over estimated speedup ceiling on both evaluated platforms.
			Throughput Speedup	Linear-Block Speedup
Platform	Method	Baseline	Realized	Estimated	Gap	Realized	Estimated	Gap
Ice Lake	Smooth-Dynamic	FP32	2.22×	3.15×	29.59%	3.59×	4.00×	10.22%
Ice Lake	Smooth-Static	FP32	2.58×	3.15×	18.17%	3.87×	4.00×	3.21%
Granite Rapids	Smooth-Dynamic	FP32	5.02×	7.06×	28.89%	9.92×	11.13×	10.89%
Granite Rapids	Smooth-Static	FP32	5.82×	7.06×	17.56%	8.67×	11.13×	22.12%
Granite Rapids	Smooth-Dynamic	BF16	1.37×	1.53×	10.41%	1.84×	1.91×	3.58%
Granite Rapids	Smooth-Static	BF16	1.59×	1.53×	-3.98%	1.61×	1.91×	15.64%

Table 15: Accuracy results on Granite Rapids.
Model	Metric	FP32	BF16	Smooth-Static	Alpha	Loss	Smooth-Dynamic	Alpha	Loss
BERT-large	SQuAD F1	92.64	92.67	92.66	0.6	0.0%	92.63	0.6	0.0%
BERT-large	SQuAD EM	86.41	86.45	86.49	0.6	-0.1%	86.44	0.6	0.0%
BERT-large	MultiNLI Accuracy	64.55	64.58	64.73	0.7	-0.3%	64.52	0.6	0.0%
DistilBERT	SQuAD F1	86.39	86.41	86.00	0.7	0.5%	86.40	0.6	0.0%
DistilBERT	SQuAD EM	79.02	79.04	78.49	0.7	0.7%	79.04	0.6	0.0%
DistilBERT	MultiNLI Accuracy	82.21	82.16	82.26	0.8	-0.1%	82.17	0.6	0.0%
XLM-RoBERTa	SQuAD F1	76.43	76.34	76.47	0.55	-0.1%	76.59	0.6	-0.2%
XLM-RoBERTa	SQuAD EM	69.95	69.89	70.04	0.55	-0.1%	70.30	0.6	-0.5%
XLM-RoBERTa	MultiNLI Accuracy	82.26	82.23	82.31	0.7	-0.1%	82.26	0.6	0.0%

Why it matters

Many companies still rely on smaller, cheaper-to-run NLP models for everyday tasks like search ranking and text classification, and most of them deploy on ordinary CPUs rather than GPUs. Making these speedups available directly inside standard PyTorch, without extra plugins, lowers the barrier for any team to get several-times-faster, cheaper inference with almost no code changes.

Terms in this paper

INT8 quantization · Storing and computing numbers using 8-bit integers instead of 32-bit floats, which is faster and uses less memory but can reduce accuracy.
SmoothQuant · A technique that rebalances numerical scale between activations and weights so INT8 quantization causes less accuracy loss.
TorchAO · PyTorch's official library for applying quantization and other low-precision techniques to models.
TorchInductor · The compiler backend in PyTorch 2 that optimizes and generates fast code for a model's computation graph.
GEMM · General matrix multiplication, the core repeated calculation inside neural network layers.
AVX512_VNNI / AMX · Special instruction sets built into Intel Xeon CPUs that speed up integer math used in INT8 computation.
roofline model · A performance analysis method that predicts whether a computation is limited by processor speed or by memory bandwidth.

Original abstract (English)

Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling

Authors · Weiwen Xia, Yuxin Cui, E Cao

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Weiwen Xia et al., arXiv:2608.18182, CC BY 4.0