Figure 1: SmoothQuant workflow built on TorchAO and TorchInductor. TorchAO prepares and quantizes the model, TorchInductor fuses INT8 subgraphs, and the CPP backend selects between oneDNN and template-based GEMM kernels. C++ code is generated and compiled for deployment.
Table 1: Hardware platforms used for baseline collection.
Xeon Generation
Codename
Model
Physical Cores
Instruction Support
3rd Gen
Ice Lake
8358
64
AVX512F, AVX512_VNNI
6th Gen
Granite Rapids
6980P
256
AVX512F, AVX512_VNNI, AMX
Table 2: Baseline collection setup.
Item
Value
Model
BERT-large
Input shape
Batch size 1, sequence length 256
Precision modes
FP32 on both platforms; BF16 on Granite Rapids
Launch strategy
Multi-instance with weight-sharing on each socket
Cores per instance
4
Number of instances
Number of physical cores / 4
Deployment path
AOTI enabled
Input data
Synthetic
Table 3: Baseline hotspot breakdown for the BERT-large analysis case under the same multi-instance setup used for throughput benchmarking. Operator shares are aggregated from representative profiler logs. The linear-block consists of GEMM and fused post-operations such as bias addition, activation, and residual addition.
Baseline
Linear-Block
Attention
LayerNorm
Other
FP32 on Ice Lake
88.5%
10.4%
0.6%
0.5%
FP32 on Granite Rapids
86.9%
11.8%
0.8%
0.5%
BF16 on Granite Rapids
72.70%
19.75%
5.99%
1.56%
Table 4: Estimated peak compute and measured memory bandwidth per socket, balance point for different data types on the evaluated platforms.
Platform
FP32 Throughput (TFLOPS)
BF16 Throughput (TFLOPS)
INT8 Throughput (TOPS)
Memory Bandwidth (GB/s)
FP32 Balance Point (FLOPs/byte)
BF16 Balance Point (FLOPs/byte)
INT8 Balance Point (OPs/byte)
Ice Lake
6.76
N/A
27.03
175.00
38.63
N/A
154.46
Granite Rapids
19.66
262.14
524.29
700.00
28.09
374.49
748.99
Table 5: BERT-large GEMM shapes and their arithmetic intensity.
Arithmetic Intensity (ops/byte)
GEMM Shape M × K × N
Occurrences per forward pass
FP32
BF16
INT8
256×1024×1024
96
85.33
146.29
227.56
256×1024×4096
24
97.52
163.84
248.24
256×4096×1024
24
97.52
186.18
341.33
Table 6: Effective memory bandwidth and balance point for INT8 GEMM on Granite Rapids.
GEMM Shape
Effective Memory Bandwidth (GB/s)
Arithmetic Intensity (OPs/byte)
Balance Point (OPs/byte)
256×1024×1024
807.69
227.56
649.12
256×1024×4096
819.15
248.24
640.04
256×4096×1024
875.00
341.33
599.19
Table 7: Estimated GEMM throughput and speedup on Granite Rapids.
GEMM Shape
FP32 Throughput (TFLOPS)
BF16 Throughput (TFLOPS)
INT8 Throughput (TOPS)
INT8 vs FP32 Speedup
INT8 vs BF16 Speedup
256×1024×1024
19.66
102.40
183.79
9.35×
1.79×
256×1024×4096
19.66
114.69
203.35
10.34×
1.77×
256×4096×1024
19.66
130.33
298.67
15.19×
2.29×
Weighted Average
19.66
114.69
218.87
11.13×
1.91×
Table 8: Models used for the performance benchmark.
Model
Model ID
BERT-large
bert-large-uncased
DistilBERT
distilbert-base-uncased
XLM-RoBERTa
xlm-roberta-base
Table 9: Task-finetuned checkpoints used for accuracy evaluation. The model marked by * is finetuned on SQuAD 2.0 but still workable for the test on SQuAD 1.1.
Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling
