May 20, 2026 — Today at the Annual Conference on Machine Learning and Systems (MLSys 2026), I presented our paper “Automated Algorithm Design for Auto-Tuning Optimizers”. It was wonderful to see so many people attend the talk and engage in discussions afterwards. In this paper, Niki van Stein, Ben van Werkhoven and I explore whether LLMs can automatically generate optimization algorithms for HPC auto-tuning - and show they can outperform even our best human-designed algorithms.
The best performing LLM-generated algorithms are now included in Kernel Tuner to allow all Kernel Tuner users to benefit from our work.
Abstract
Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning.
Citation
F.J. Willemsen, N. van Stein, B. van Werkhoven “Automated Algorithm Design for Auto-Tuning Optimizers” Ninth Annual Conference on Machine Learning and Systems (MLSys 2026), Bellevue, WA, May 18-22, 2026 preprint
