Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX
arXiv:2608.173792026-08-17
AI models can write GPU low-level code, but only get it half right
PTXBench is a new benchmark that tests whether AI models can directly use NVIDIA's lowest-level GPU instruction set (PTX) to write fast kernels on H100 and B200 GPUs. Testing several frontier models on matrix multiplication (GEMM) and attention workloads, the researchers found models often succeed on simpler forward-pass tasks but fail much more often on complex backward attention, and even when a model manages to execute the required low-level instructions, that doesn't guarantee competitive speed. The team also fine-tuned an open model, Qwen3.6-27B, and found gains were inconsistent, depending heavily on training data balance and the quality of the teacher model used to generate corrections.
METAL MEDIA explanatory visual
AI models can write GPU low-level code, but only get it half right
01Introduced PTXBench, a benchmark that measures whether LLMs can use architecture-specific PTX (NVIDIA's lowest-level programmable GPU interface) to write efficient kernels
02Models are given GPU architecture knowledge and allowed multiple turns to revise CUDA kernels, while the benchmark separately checks correctness, whether the target low-level instructions actually run, and speed compared to frontier libraries like cuBLAS
03Testing Gemini 3.1 Pro, Claude Opus 4.8, GLM-5.2, and Qwen3.6-27B showed strong performance on forward workloads but a sharp drop in success on complex backward attention tasks; executing the right instructions didn't always translate to faster kernels
04Using a method called Fixit, the team fine-tuned Qwen3.6-27B on collected failures paired with teacher-corrected fixes and explanations, improving some tasks but generalizing poorly to new problem variants
05Data coverage and balance across problem types, plus the skill of the teacher model providing corrections, mattered more than simply having more training examples
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Introduced PTXBench, a benchmark that measures whether LLMs can use architecture-specific PTX (NVIDIA's lowest-level programmable GPU interface) to write efficient kernels
Models are given GPU architecture knowledge and allowed multiple turns to revise CUDA kernels, while the benchmark separately checks correctness, whether the target low-level instructions actually run, and speed compared to frontier libraries like cuBLAS
Testing Gemini 3.1 Pro, Claude Opus 4.8, GLM-5.2, and Qwen3.6-27B showed strong performance on forward workloads but a sharp drop in success on complex backward attention tasks; executing the right instructions didn't always translate to faster kernels
Using a method called Fixit, the team fine-tuned Qwen3.6-27B on collected failures paired with teacher-corrected fixes and explanations, improving some tasks but generalizing poorly to new problem variants
Data coverage and balance across problem types, plus the skill of the teacher model providing corrections, mattered more than simply having more training examples
Figure 1: Overview of the PTXBench benchmark and adaptation workflow.
Table 1: Target-instruction and unrestricted turn correctness rates (%) on H100 and B200.
GPU
Model
GEMM
MHA-Fwd
MHA-Fwd-Causal
MHA-Bwd
MHA-Bwd-Causal
H100
Gemini 3.1 Pro
33.3 / 60.4 / 56.2 (33.3 / 62.5 / 59.4)
33.3 / 39.6 / 45.8 (33.3 / 39.6 / 45.8)
25.0 / 52.1 / 55.2 (25.0 / 52.1 / 55.2)
8.3 / 33.3 / 38.5 (8.3 / 33.3 / 38.5)
– / 22.9 / 25.0 (8.3 / 25.0 / 26.0)
Claude Opus 4.8
91.7 / 95.8 / 94.8 (91.7 / 95.8 / 94.8)
50.0 / 81.2 / 90.6 (66.7 / 89.6 / 94.8)
50.0 / 77.1 / 75.0 (83.3 / 89.6 / 82.3)
8.3 / 62.5 / 79.2 (50.0 / 83.3 / 89.6)
– / 22.9 / 44.8 (25.0 / 41.7 / 59.4)
GLM-5.2
33.3 / 60.4 / 61.5 (33.3 / 62.5 / 62.5)
16.7 / 8.3 / 8.3 (16.7 / 14.6 / 15.6)
– / 8.3 / 10.4 (8.3 / 16.7 / 17.7)
8.3 / 6.2 / 5.2 (8.3 / 18.8 / 16.7)
– (– / 22.9 / 15.6)
Qwen3.6-27B
–
–
–
–
–
B200
Gemini 3.1 Pro
8.3 / 47.9 / 45.8 (8.3 / 47.9 / 45.8)
– / 12.5 / 15.6 (– / 12.5 / 15.6)
– / 8.3 / 10.4 (8.3 / 12.5 / 12.5)
– / 2.1 / 2.1 (– / 2.1 / 2.1)
– (– / 2.1 / 1.0)
Claude Opus 4.8
25.0 / 64.6 / 80.2 (75.0 / 81.2 / 88.5)
– / 20.8 / 38.5 (83.3 / 87.5 / 86.5)
– / 16.7 / 32.3 (91.7 / 77.1 / 83.3)
– / 6.2 / 10.4 (83.3 / 89.6 / 91.7)
– (25.0 / 52.1 / 68.8)
GLM-5.2
16.7 / 27.1 / 28.1 (33.3 / 37.5 / 34.4)
– / – / 1.0 (33.3 / 22.9 / 26.0)
8.3 / 2.1 / 1.0 (33.3 / 22.9 / 25.0)
– (8.3 / 25.0 / 30.2)
– (16.7 / 18.8 / 22.9)
Qwen3.6-27B
– (– / – / 1.0)
–
–
–
–
Figure 2: FastpInst. on H100 (top) and B200 (bottom).
Table 2: Model release dates, knowledge cutoffs, and estimated calendar lag from the release of Hopper PTX ISA 8.0 (Dec. 2022) and Blackwell PTX ISA 8.7 (Jan. 2025) [26, 29]. Lags use monthly granularity, from PTX release to disclosed cutoff; otherwise, model release dates provide upper bounds.
Model
Model release
Knowledge cutoff
Lag after PTX release (months)
Hopper
Blackwell
Gemini 3.1 Pro
Feb. 2026
Jan. 2025
25
≈0
Claude Opus 4.8
May 2026
Jan. 2026
37
12
GLM-5.2
June 2026
Not disclosed
≤42
≤17
Qwen3.6-27B
Apr. 2026
Not disclosed
≤40
≤15
Figure 3: Gemini 3.1 Pro Fastp distributions for Triton and CUDA-PTX on H100 and B200.
Table 3: Ablation of architecture-specific prompt knowledge for Gemini 3.1 Pro.
Figure 4: Training data recipes. Pie area is proportional to record count, and slices show the fraction drawn from each problem. The top shows labels for the checkpoints; the bottom line lists training formats and reasoning teachers.
Table 4: Target-instruction and unrestricted turn correctness rates (%) under SFT and prompt-time supervision.
Condition
MHA-Fwd
MHA-Fwd-Causal
MHA-Bwd
MHA-Bwd-Causal
Qwen3.6-27B w/o expert guidance
–
–
–
–
Qwen3.6-27B
–
–
–
–
Qwen3.6-27B-s1 w/o expert guidance
–
– / – / 4.2 (– / – / 4.2)
– / – / 4.2 (– / – / 4.2)
– / 4.2 / 3.1 (– / 4.2 / 3.1)
Qwen3.6-27B-s1
16.7 / 16.7 / 19.8 (16.7 / 16.7 / 19.8)
– / – / 3.1 (– / – / 3.1)
– / 2.1 / 4.2 (– / 2.1 / 4.2)
– / 2.1 / 4.2 (– / 2.1 / 5.2)
Qwen3.6-27B + retrieved repair notes
–
–
–
–
Qwen3.6-27B + retrieved repair notes and fixed kernel
– / 29.2 / 29.2 (– / 29.2 / 29.2)
– / 20.8 / 27.1 (– / 20.8 / 27.1)
– / 14.6 / 18.8 (– / 14.6 / 20.8)
– / 20.8 / 33.3 (– / 25.0 / 37.5)
Figure 5: SFT training data recipe comparison (complete results in Appendix Tables 8 and 9).
Table 5: Selected SASS families for target instruction measurement.
GPU
Tag
Selected SASS family
H100 (Hopper)
H
*GMMA tensor compute; UTMALDG, UTMASTG, and UTMAREDG TMA payload transfer
B200 (Blackwell)
B
UTC*, LDTM, and STTM from the TCGEN05 tensor pathway
Figure 6: Correctness and speedup of Qwen3.6-27B-s1 on the training and held-out problems.
Table 6: Token counts for architecture-specific prompt components.
GPU
Component
Tokens
H100
Architecture parameter
259
Template functions
18,601
Architecture contract
3,454
Total
22,314
B200
Architecture parameter
413
Template functions
18,871
Architecture contract
9,531
Total
28,815
Figure 7: How Fixit SFT changes reasoning length and error types across turns.
Table 7: Best speedup with target instruction execution on H100 and B200.
GPU
Model
GEMM
MHA-Fwd
MHA-Fwd-Causal
MHA-Bwd
MHA-Bwd-Causal
H100
Gemini 3.1 Pro
0.687 / 0.934 / 0.962 (0.687 / 0.934 / 0.962)
0.555 / 0.730 / 0.730 (0.555 / 0.730 / 0.730)
0.614 / 0.651 / 0.768 (0.614 / 0.651 / 0.768)
0.206 / 0.375 / 0.515 (0.206 / 0.375 / 0.515)
– / 0.634 / 0.639 (0.065 / 0.634 / 0.639)
Claude Opus 4.8
0.770 / 0.968 / 0.976 (0.770 / 0.968 / 0.976)
0.759 / 0.770 / 0.839 (0.759 / 0.770 / 0.839)
0.758 / 0.806 / 0.806 (0.758 / 0.806 / 0.806)
0.300 / 0.440 / 0.489 (0.300 / 0.440 / 0.489)
– / 0.499 / 0.499 (0.058 / 0.499 / 0.499)
GLM-5.2
0.447 / 0.692 / 0.692 (0.447 / 0.692 / 0.692)
0.407 / 0.470 / 0.607 (0.407 / 0.470 / 0.607)
– / 0.471 / 0.533 (0.015 / 0.471 / 0.533)
0.316 / 0.437 / 0.437 (0.316 / 0.437 / 0.437)
– (– / 0.101 / 0.101)
Qwen3.6-27B
–
–
–
–
–
B200
Gemini 3.1 Pro
0.273 / 0.680 / 0.892 (0.273 / 0.680 / 0.892)
– / 0.280 / 0.280 (– / 0.280 / 0.280)
– / 0.206 / 0.248 (0.013 / 0.206 / 0.248)
– / 0.087 / 0.133 (– / 0.087 / 0.133)
– (– / 0.015 / 0.015)
Claude Opus 4.8
0.782 / 1.012 / 1.012 (0.782 / 1.012 / 1.012)
– / 0.253 / 0.300 (0.110 / 0.253 / 0.300)
– / 0.232 / 0.269 (0.042 / 0.232 / 0.269)
– / 0.069 / 0.149 (0.022 / 0.136 / 0.155)
– (0.023 / 0.088 / 0.116)
GLM-5.2
0.162 / 0.632 / 0.632 (0.162 / 0.632 / 0.632)
– / – / 0.027 (0.024 / 0.024 / 0.035)
0.098 / 0.098 / 0.098 (0.098 / 0.098 / 0.098)
– (0.019 / 0.030 / 0.040)
– (0.018 / 0.034 / 0.034)
Qwen3.6-27B
– (– / – / 0.006)
–
–
–
–
Figure 8: Cross-language transfer of Fixit SFT from CUDA-PTX to Triton on Hopper.
Table 8: Turn correctness rates for the five-problem SFT evaluation.
SFT-ed Model Label
GEMM
MHA-Fwd
MHA-Fwd-Causal
MHA-Bwd
MHA-Bwd-Causal
Qwen3.6-27B-s0
–
– / 6.2 / 5.2 (– / 6.2 / 5.2)
– / 2.1 / 1.0 (– / 2.1 / 1.0)
–
– / – / 1.0 (– / – / 1.0)
Qwen3.6-27B-s1
25.0 / 14.6 / 13.5 (25.0 / 14.6 / 13.5)
16.7 / 16.7 / 19.8 (16.7 / 16.7 / 19.8)
– / – / 3.1 (– / – / 3.1)
– / 2.1 / 4.2 (– / 2.1 / 4.2)
– / 2.1 / 4.2 (– / 2.1 / 5.2)
Qwen3.6-27B-s2
– / 2.1 / 2.1 (– / 2.1 / 2.1)
– / 2.1 / 5.2 (– / 2.1 / 5.2)
– / – / 1.0 (– / – / 1.0)
– / – / 1.0 (– / – / 1.0)
–
Qwen3.6-27B-s3
– / 2.1 / 3.1 (– / 2.1 / 3.1)
– / 4.2 / 2.1 (– / 4.2 / 2.1)
– / 6.2 / 4.2 (– / 6.2 / 4.2)
– / 4.2 / 4.2 (– / 4.2 / 4.2)
–
Qwen3.6-27B-s4
8.3 / 8.3 / 10.4 (8.3 / 8.3 / 10.4)
– / – / 4.2 (– / – / 4.2)
– / 4.2 / 2.1 (– / 4.2 / 2.1)
– / – / 1.0 (– / – / 1.0)
–
Qwen3.6-27B-s5
16.7 / 14.6 / 12.5 (16.7 / 14.6 / 12.5)
– / 2.1 / 4.2 (– / 2.1 / 4.2)
– / 2.1 / 3.1 (– / 2.1 / 3.1)
– / – / 5.2 (– / – / 5.2)
– / – / 4.2 (– / – / 4.2)
Qwen3.6-27B-s6
8.3 / 2.1 / 1.0 (8.3 / 2.1 / 1.0)
–
–
–
–
Figure 9: Turn-level error-state transitions for GEMM.
Table 9: Best speedups for the five-problem SFT evaluation.
SFT-ed Model Label
GEMM
MHA-Fwd
MHA-Fwd-Causal
MHA-Bwd
MHA-Bwd-Causal
Qwen3.6-27B-s0
–
– / 0.589 / 0.589 (– / 0.589 / 0.589)
– / 0.644 / 0.644 (– / 0.644 / 0.644)
–
– / – / 0.209 (– / – / 0.209)
Qwen3.6-27B-s1
0.303 / 0.303 / 0.340 (0.303 / 0.303 / 0.340)
0.556 / 0.556 / 0.565 (0.556 / 0.556 / 0.565)
– / – / 0.395 (– / – / 0.395)
– / 0.380 / 0.389 (– / 0.380 / 0.389)
– / 0.192 / 0.199 (– / 0.192 / 0.199)
Qwen3.6-27B-s2
– / 0.209 / 0.211 (– / 0.209 / 0.211)
– / 0.548 / 0.548 (– / 0.548 / 0.548)
– / – / 0.315 (– / – / 0.315)
– / – / 0.296 (– / – / 0.296)
–
Qwen3.6-27B-s3
– / 0.274 / 0.280 (– / 0.274 / 0.280)
– / 0.465 / 0.465 (– / 0.465 / 0.465)
– / 0.388 / 0.388 (– / 0.388 / 0.388)
– / 0.494 / 0.494 (– / 0.494 / 0.494)
–
Qwen3.6-27B-s4
0.276 / 0.373 / 0.373 (0.276 / 0.373 / 0.373)
– / – / 0.452 (– / – / 0.452)
– / 0.383 / 0.383 (– / 0.383 / 0.383)
– / – / 0.199 (– / – / 0.199)
–
Qwen3.6-27B-s5
0.325 / 0.446 / 0.446 (0.325 / 0.446 / 0.446)
– / 0.425 / 0.573 (– / 0.425 / 0.573)
– / 0.241 / 0.246 (– / 0.241 / 0.246)
– / – / 0.295 (– / – / 0.295)
– / – / 0.246 (– / – / 0.246)
Qwen3.6-27B-s6
0.073 / 0.073 / 0.073 (0.073 / 0.073 / 0.073)
–
–
–
–
Figure 10: FastpInst. under SFT and prompt-time supervision.
Table 10: Training data recipes.
SFT-ed Model Label
Config
Template
Reasoning Synthesizer
Record Count
Qwen3.6-27B-s0
8ops-Extended
KernelGen
GLM-5.2
494
Qwen3.6-27B-s1
4ops
Fixit
GLM-5.2
158
Qwen3.6-27B-s2
4ops-Extended
Fixit
GLM-5.2
259
Qwen3.6-27B-s3
8ops-Extended
Fixit
GLM-5.2
406
Qwen3.6-27B-s4
8ops-Post-balanced
Fixit
GLM-5.2
170
Qwen3.6-27B-s5
8ops-Pre-balanced
Fixit
GLM-5.2
258
Qwen3.6-27B-s6
8ops-Pre-balanced
Fixit
Qwen3.6-27B
258
Figure 11: Detailed execution infrastructure supporting PTXBench. MiniPTXAgent compiles generated CUDA–PTX locally and sends successfully compiled candidates to an isolated GPU profiling service for sanitization, evaluation, and optional diagnostics. A reliability monitor pauses dispatch, restarts unhealthy service state, and excludes affected turns from trajectories. After a restart, it also excludes thermally abnormal GPUs because throttling distorted latency by over 15% in our measurements [35].
Why it matters
GPU kernel optimization directly determines how fast and cheap AI training and inference can be, but writing this low-level code by hand requires rare expert skill. This work is one of the first controlled tests of whether AI can genuinely exploit new hardware features, and what kind of training actually helps it improve.
Figure 12: Evaluation of workload state caching, generation-profiling pipelining, and GPU sharing.
GEMM · General matrix multiplication, a standard workload used to benchmark GPU performance
SFT (supervised fine-tuning) · Further training a model on labeled example solutions
LoRA · A lightweight fine-tuning method that updates only a small part of a model instead of the whole thing
SASS · NVIDIA's native machine code that PTX instructions compile down to when actually run on the GPU
Original abstract (English)
We introduce PTXBench, a benchmark for evaluating and adapting large language models (LLMs) to use architecture-specific PTX for GPU kernel optimization. PTXBench measures functional correctness, whether selected target instructions execute at runtime, and speedup over frontier libraries across GEMM and attention workloads on H100 and B200 GPUs. Our evaluation shows that architecture-specific PTX capability remains uneven: success rates fall substantially on complex attention backward workloads, and executing the target instructions does not necessarily translate into competitive performance. No evaluated model consistently matches frontier libraries across the suite. We further adapt Qwen3.6-27B using supervised fine-tuning. Repair-conditioned training improves several tasks, but generalization remains uneven; data coverage, balance, and the quality of the reasoning teacher matter in addition to dataset size. PTXBench provides an auditable testbed for measuring and improving LLMs' ability to exploit evolving GPU architectures.
![Figure 11: Detailed execution infrastructure supporting PTXBench. MiniPTXAgent compiles generated CUDA–PTX locally and sends successfully compiled candidates to an isolated GPU profiling service for sanitization, evaluation, and optional diagnostics. A reliability monitor pauses dispatch, restarts unhealthy service state, and excludes affected turns from trajectories. After a restart, it also excludes thermally abnormal GPUs because throttling distorted latency by over 15% in our measurements [35].](https://media.metallab.ai/papers/2608.17379/f10.png)
