PTXBench tests whether LLMs can really tune GPU kernels

A new benchmark called PTXBench is testing whether language models can do more than write plausible GPU optimization code.

The benchmark focuses on PTX, Nvidia’s low-level parallel thread execution language, across GEMM and attention workloads on H100 and B200 GPUs. It measures not only functional correctness, but also whether selected target instructions actually run and whether the result improves speed compared with frontier libraries.

The results are mixed. The authors report that architecture-specific PTX ability is uneven, especially on complex attention backward workloads. Even when a model uses the intended instructions, that does not guarantee competitive performance.

The team also fine-tuned Qwen3.6-27B and found that repair-conditioned training helped on some tasks, but did not solve generalization. For developers, the benchmark points to a practical limit: AI coding agents may assist with kernel work, but current models still need strong tests, hardware-aware evaluation, and expert review before performance claims mean much.