Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack
arXiv:2608.181822026-08-20
Intel engineers made BERT-style models run up to 5.8x faster on server CPUs, using only PyTorch's built-in tools
BERT-family models like BERT and DistilBERT are still heavily used in industry for tasks like classification and search, even though everyone talks about large language models now. This work plugs SmoothQuant, a technique that squeezes model numbers into 8-bit integers (INT8) without losing much accuracy, directly into PyTorch's own compiler stack instead of requiring separate third-party tools. The result is up to 5.8x faster processing on Intel Xeon server chips with barely any accuracy loss.
METAL MEDIA explanatory visual
Intel engineers made BERT-style models run up to 5.8x faster on server CPUs, using only PyTorch's built-in tools
01Integrated SmoothQuant (a method that shifts numerical 'difficulty' from activations to weights before compressing to INT8) into TorchAO, PyTorch's native quantization library, so users don't need external tools.
02Added fusion passes in TorchInductor (PyTorch's compiler backend) to merge INT8 matrix-multiplication operations with scaling and bias steps into a single efficient operator, avoiding wasted time converting data layouts at runtime.
03Built custom low-level kernels using AVX512_VNNI and AMX (special CPU instruction sets for fast integer math) so both older Ice Lake chips and newer Granite Rapids chips get the best available speed.
04Devised a workaround (s8s8-to-u8s8 transformation) so INT8 quantization still works efficiently on Ice Lake CPUs that lack certain instruction support.
05Benchmarked on BERT-large, DistilBERT, and XLM-RoBERTa: throughput improved 1.9x-2.6x on Ice Lake and 4.2x-5.8x on Granite Rapids versus FP32 (32-bit floating point, the default precision), with less than 1% accuracy loss on SQuAD and MultiNLI tasks.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Integrated SmoothQuant (a method that shifts numerical 'difficulty' from activations to weights before compressing to INT8) into TorchAO, PyTorch's native quantization library, so users don't need external tools.
Added fusion passes in TorchInductor (PyTorch's compiler backend) to merge INT8 matrix-multiplication operations with scaling and bias steps into a single efficient operator, avoiding wasted time converting data layouts at runtime.
Built custom low-level kernels using AVX512_VNNI and AMX (special CPU instruction sets for fast integer math) so both older Ice Lake chips and newer Granite Rapids chips get the best available speed.
Devised a workaround (s8s8-to-u8s8 transformation) so INT8 quantization still works efficiently on Ice Lake CPUs that lack certain instruction support.
Benchmarked on BERT-large, DistilBERT, and XLM-RoBERTa: throughput improved 1.9x-2.6x on Ice Lake and 4.2x-5.8x on Granite Rapids versus FP32 (32-bit floating point, the default precision), with less than 1% accuracy loss on SQuAD and MultiNLI tasks.
Figure 1: SmoothQuant workflow built on TorchAO and TorchInductor. TorchAO prepares and quantizes the model, TorchInductor fuses INT8 subgraphs, and the CPP backend selects between oneDNN and template-based GEMM kernels. C++ code is generated and compiled for deployment.
Table 1: Hardware platforms used for baseline collection.
Xeon Generation
Codename
Model
Physical Cores
Instruction Support
3rd Gen
Ice Lake
8358
64
AVX512F, AVX512_VNNI
6th Gen
Granite Rapids
6980P
256
AVX512F, AVX512_VNNI, AMX
Table 2: Baseline collection setup.
Item
Value
Model
BERT-large
Input shape
Batch size 1, sequence length 256
Precision modes
FP32 on both platforms; BF16 on Granite Rapids
Launch strategy
Multi-instance with weight-sharing on each socket
Cores per instance
4
Number of instances
Number of physical cores / 4
Deployment path
AOTI enabled
Input data
Synthetic
Table 3: Baseline hotspot breakdown for the BERT-large analysis case under the same multi-instance setup used for throughput benchmarking. Operator shares are aggregated from representative profiler logs. The linear-block consists of GEMM and fused post-operations such as bias addition, activation, and residual addition.
Baseline
Linear-Block
Attention
LayerNorm
Other
FP32 on Ice Lake
88.5%
10.4%
0.6%
0.5%
FP32 on Granite Rapids
86.9%
11.8%
0.8%
0.5%
BF16 on Granite Rapids
72.70%
19.75%
5.99%
1.56%
Table 4: Estimated peak compute and measured memory bandwidth per socket, balance point for different data types on the evaluated platforms.
Platform
FP32 Throughput (TFLOPS)
BF16 Throughput (TFLOPS)
INT8 Throughput (TOPS)
Memory Bandwidth (GB/s)
FP32 Balance Point (FLOPs/byte)
BF16 Balance Point (FLOPs/byte)
INT8 Balance Point (OPs/byte)
Ice Lake
6.76
N/A
27.03
175.00
38.63
N/A
154.46
Granite Rapids
19.66
262.14
524.29
700.00
28.09
374.49
748.99
Table 5: BERT-large GEMM shapes and their arithmetic intensity.
Arithmetic Intensity (ops/byte)
GEMM Shape M × K × N
Occurrences per forward pass
FP32
BF16
INT8
256×1024×1024
96
85.33
146.29
227.56
256×1024×4096
24
97.52
163.84
248.24
256×4096×1024
24
97.52
186.18
341.33
Table 6: Effective memory bandwidth and balance point for INT8 GEMM on Granite Rapids.
GEMM Shape
Effective Memory Bandwidth (GB/s)
Arithmetic Intensity (OPs/byte)
Balance Point (OPs/byte)
256×1024×1024
807.69
227.56
649.12
256×1024×4096
819.15
248.24
640.04
256×4096×1024
875.00
341.33
599.19
Table 7: Estimated GEMM throughput and speedup on Granite Rapids.
GEMM Shape
FP32 Throughput (TFLOPS)
BF16 Throughput (TFLOPS)
INT8 Throughput (TOPS)
INT8 vs FP32 Speedup
INT8 vs BF16 Speedup
256×1024×1024
19.66
102.40
183.79
9.35×
1.79×
256×1024×4096
19.66
114.69
203.35
10.34×
1.77×
256×4096×1024
19.66
130.33
298.67
15.19×
2.29×
Weighted Average
19.66
114.69
218.87
11.13×
1.91×
Table 8: Models used for the performance benchmark.
Model
Model ID
BERT-large
bert-large-uncased
DistilBERT
distilbert-base-uncased
XLM-RoBERTa
xlm-roberta-base
Table 9: Task-finetuned checkpoints used for accuracy evaluation. The model marked by * is finetuned on SQuAD 2.0 but still workable for the test on SQuAD 1.1.
Table 10: Quantization methods used in the experiments.
Method
Short Name
No quantization, no AMP
FP32
No quantization, with AMP
BF16
SmoothQuant, static quantization
Smooth-Static
SmoothQuant, dynamic quantization
Smooth-Dynamic
Table 11: INT8 speedup over FP32 on Ice Lake.
Model
Method
Throughput Speedup
Linear-Block Speedup
BERT-large
Smooth-Dynamic
2.22×
3.59×
BERT-large
Smooth-Static
2.58×
3.87×
DistilBERT
Smooth-Dynamic
1.90×
3.31×
DistilBERT
Smooth-Static
2.30×
3.35×
XLM-RoBERTa
Smooth-Dynamic
1.86×
3.05×
XLM-RoBERTa
Smooth-Static
2.17×
3.71×
Table 12: INT8 speedup over FP32 on Granite Rapids.
Model
Method
Throughput Speedup
Linear-Block Speedup
BERT-large
Smooth-Dynamic
5.02×
9.92×
BERT-large
Smooth-Static
5.82×
8.67×
DistilBERT
Smooth-Dynamic
4.24×
7.60×
DistilBERT
Smooth-Static
5.37×
8.84×
XLM-RoBERTa
Smooth-Dynamic
4.17×
7.58×
XLM-RoBERTa
Smooth-Static
5.41×
8.71×
Table 13: INT8 speedup over BF16 on Granite Rapids.
Model
Method
Throughput Speedup
Linear-Block Speedup
BERT-large
Smooth-Dynamic
1.37×
1.84×
BERT-large
Smooth-Static
1.59×
1.61×
DistilBERT
Smooth-Dynamic
0.98×
1.45×
DistilBERT
Smooth-Static
1.24×
1.69×
XLM-RoBERTa
Smooth-Dynamic
0.99×
1.50×
XLM-RoBERTa
Smooth-Static
1.29×
1.67×
Table 14: Realized speedup of BERT-large over estimated speedup ceiling on both evaluated platforms.
Throughput Speedup
Linear-Block Speedup
Platform
Method
Baseline
Realized
Estimated
Gap
Realized
Estimated
Gap
Ice Lake
Smooth-Dynamic
FP32
2.22×
3.15×
29.59%
3.59×
4.00×
10.22%
Ice Lake
Smooth-Static
FP32
2.58×
3.15×
18.17%
3.87×
4.00×
3.21%
Granite Rapids
Smooth-Dynamic
FP32
5.02×
7.06×
28.89%
9.92×
11.13×
10.89%
Granite Rapids
Smooth-Static
FP32
5.82×
7.06×
17.56%
8.67×
11.13×
22.12%
Granite Rapids
Smooth-Dynamic
BF16
1.37×
1.53×
10.41%
1.84×
1.91×
3.58%
Granite Rapids
Smooth-Static
BF16
1.59×
1.53×
-3.98%
1.61×
1.91×
15.64%
Table 15: Accuracy results on Granite Rapids.
Model
Metric
FP32
BF16
Smooth-Static
Alpha
Loss
Smooth-Dynamic
Alpha
Loss
BERT-large
SQuAD F1
92.64
92.67
92.66
0.6
0.0%
92.63
0.6
0.0%
BERT-large
SQuAD EM
86.41
86.45
86.49
0.6
-0.1%
86.44
0.6
0.0%
BERT-large
MultiNLI Accuracy
64.55
64.58
64.73
0.7
-0.3%
64.52
0.6
0.0%
DistilBERT
SQuAD F1
86.39
86.41
86.00
0.7
0.5%
86.40
0.6
0.0%
DistilBERT
SQuAD EM
79.02
79.04
78.49
0.7
0.7%
79.04
0.6
0.0%
DistilBERT
MultiNLI Accuracy
82.21
82.16
82.26
0.8
-0.1%
82.17
0.6
0.0%
XLM-RoBERTa
SQuAD F1
76.43
76.34
76.47
0.55
-0.1%
76.59
0.6
-0.2%
XLM-RoBERTa
SQuAD EM
69.95
69.89
70.04
0.55
-0.1%
70.30
0.6
-0.5%
XLM-RoBERTa
MultiNLI Accuracy
82.26
82.23
82.31
0.7
-0.1%
82.26
0.6
0.0%
Why it matters
Many companies still rely on smaller, cheaper-to-run NLP models for everyday tasks like search ranking and text classification, and most of them deploy on ordinary CPUs rather than GPUs. Making these speedups available directly inside standard PyTorch, without extra plugins, lowers the barrier for any team to get several-times-faster, cheaper inference with almost no code changes.
Terms in this paper
INT8 quantization · Storing and computing numbers using 8-bit integers instead of 32-bit floats, which is faster and uses less memory but can reduce accuracy.
SmoothQuant · A technique that rebalances numerical scale between activations and weights so INT8 quantization causes less accuracy loss.
TorchAO · PyTorch's official library for applying quantization and other low-precision techniques to models.
TorchInductor · The compiler backend in PyTorch 2 that optimizes and generates fast code for a model's computation graph.
GEMM · General matrix multiplication, the core repeated calculation inside neural network layers.
AVX512_VNNI / AMX · Special instruction sets built into Intel Xeon CPUs that speed up integer math used in INT8 computation.
roofline model · A performance analysis method that predicts whether a computation is limited by processor speed or by memory bandwidth.
Original abstract (English)
Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling
