Kernel Forge uses agents to optimize CUDA code inside PyTorch models

A new arXiv paper introduces Kernel Forge, an agentic harness for generating and optimizing CUDA kernels inside existing PyTorch models. The goal is to reduce latency and cost in the small set of operations that dominate machine-learning runtime.

Low-level GPU optimization normally requires expert engineers. The authors say many current LLM tools are limited because they test isolated kernels, emit standalone CUDA code, or make debugging hard. Kernel Forge instead accepts unmodified PyTorch models, supports vision, diffusion, and language workloads, and provides a graphical interface for monitoring candidates and failures.

The system uses Monte Carlo Tree Search to explore multiple optimization paths rather than a single linear refinement loop. In tests on an NVIDIA DGX Spark with a GB10 GPU, it optimized 14 kernels to beat PyTorch eager mode after 50 iterations per kernel.

Reported speedups included 1.52 times on ResNet-50 adaptive average pooling, 1.70 times on Stable Diffusion group normalization, and 2.83 times on Gemma softmax. The work shows how agents may assist performance engineering, though deployment still requires careful validation.