NeurIPS 2025

Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

Paper alphaXiv Code Thread

Authors

Affiliations

Richard C. Suwandi

School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

Feng Yin

School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

Juntao Wang

School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

Renjie Li

School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen
Holonyak Micro & Nanotechnology Lab, University of Illinois at Urbana-Champaign

Tsung-Hui Chang

School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen

Sergios Theodoridis

HERON CoE for Robotics and AI-ATHENA R.C., Greece

TL;DR. A poorly chosen GP kernel slows Bayesian optimization. CAKE uses large language models as genetic operators to evolve kernels during the search, and BAKER picks which one to query next, improving hyperparameter optimization, controller tuning, and photonic chip design.

Abstract

The efficiency of Bayesian optimization (BO) depends heavily on the choice of Gaussian process (GP) kernel, which helps balance exploration and exploitation under limited evaluation budgets. Traditional BO methods often use fixed kernels or heuristic selection strategies. When the selected kernel is poorly suited to the underlying objective function, these approaches can converge slowly or produce suboptimal solutions. To address this limitation, we propose Context-Aware Kernel Evolution (CAKE), which enhances BO with large language models (LLMs). CAKE uses LLMs as crossover and mutation operators to adaptively generate and refine GP kernels based on the data observed throughout optimization. We also propose BIC-Acquisition Kernel Ranking (BAKER), which selects the most effective kernel by balancing model fit, measured by the Bayesian information criterion (BIC), with the expected improvement at each BO iteration. Extensive experiments show that CAKE-based BO consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design.

Context-Aware Kernel Evolution

CAKE begins with a small set of randomly sampled observations and a population of six base kernels: squared exponential (SE), periodic (PER), linear (LIN), rational quadratic (RQ), Matérn-3/2 (M3), and Matérn-5/2 (M5)1. The observations are added to a system prompt as few-shot examples. The prompt asks the LLM to act as a Gaussian process expert, identify patterns in the observed data, and reason about kernel structures that could explain them. This process operates entirely in context and does not fine-tune or update the LLM.

CAKE workflow: an initial population of Gaussian process kernels undergoes LLM-guided crossover, mutation, fitness evaluation, and survivor selection to produce an improved population.

Figure 1. CAKE begins with an initial population of kernels. It uses the LLM for crossover and mutation, evaluates the fitness of the proposed kernels, and advances the strongest candidates to the next generation. BAKER then selects the kernel and query for the next BO iteration.

Each candidate is a compositional kernel built through addition and multiplication, both of which preserve kernel validity2. Candidate fitness is derived from the Bayesian information criterion (BIC)3, which balances Gaussian process model fit against complexity. The fitness values are normalized to the interval [0, 1] across tasks.

  1. Condition on the current observations: at every BO iteration, CAKE updates the prompt with all input–output pairs collected so far.
  2. Crossover: five times per iteration, two parent kernels are sampled with probability proportional to fitness, drawing on the population-based search used in genetic algorithms4. The LLM combines them using addition or multiplication and explains the proposed structure.
  3. Mutation: with probability 0.7, the LLM modifies the fittest kernel by replacing one base component with another. The crossover and mutation candidates are evaluated together with the existing population.
  4. Survivor selection: CAKE retains the ten kernels with the highest normalized BIC fitness to form the next generation.
  5. Rank and query: BAKER weights each surviving kernel in proportion to exp(−BIC)a, multiplies that weight by the normalized expected improvement at the kernel’s proposed query, and selects the highest-scoring kernel–query pair. The objective is evaluated and the new observation is added before the loop repeats.

BAKER is important because the best-fitting kernel does not necessarily propose the most useful next evaluation. Its joint ranking explicitly balances model fit with the potential optimization gain of the candidate query.

Experiments

All experiments use BoTorch5, with expected improvement6 as the default acquisition function and gpt-4o-mini as the LLMb. The evaluation compares CAKE with fixed kernels, adaptive kernel selection based on Random, Utility, or BIC criteria, deep and ensemble Gaussian processes, Compositional Kernel Search (CKS)2, and Automated Bayesian Optimization (ABO)7. Shaded regions in the plots show the standard error across independent trialsc.

Hyperparameter tuning

For hyperparameter optimization, HPOBench8 provides 60 tasks drawn from 12 OpenML datasets and five model families: logistic regression, support vector machines, random forests, XGBoost, and multilayer perceptrons. Each model and dataset pair is optimized for test accuracy over 100 trials and 20 random seeds. CAKE achieves the highest average test accuracy across all five model families. Its advantage is especially clear early in the optimization process. Averaged across HPOBench, CAKE achieves 67.5% of its eventual improvement within the first quarter of the budget and more than 83% by the halfway point.

Five learning curves for logistic regression, SVM, random forest, XGBoost, and MLP. The black CAKE curve finishes above all comparison methods in every panel.

Figure 2. Average test accuracy over 20 random seeds. CAKE is shown in black, and the shaded regions indicate standard error.

Controller tuning

The controller experiments evaluate adaptation to changing environments. The robot-pushing task9 optimizes 14 controller parameters that govern two robotic hands. The lunar-landing task10 optimizes 12 parameters that map an eight-dimensional state to four actions. Both tasks run for 1,000 iterations, with results averaged over ten initial conditions. CAKE achieves the highest average reward in both tasks and converges fastest in robot pushing. For lunar landing, ABO is the only baseline that also reaches the target reward of 200, although its performance fluctuates more than CAKE across difficult environments. The fixed SE and M5 kernels tend to plateau earlier.

Reward curves for robot pushing and lunar lander. The black CAKE curve ends highest in both panels and exceeds the lunar-lander target score of 200.

Figure 3. Controller-tuning performance over 1,000 iterations, averaged across ten initial conditions.

Photonic chip design

Photonic chip design is formulated as a multi-objective black-box problem11 with five competing physical indicators: Q-factor, wavelength, lasing area, power, and divergence angle. CAKE is evaluated over 250 trials and ten random initializations using expected hypervolume improvement12. The comparison methods are Single-Task GP, Additive GP, CKS, and ABO. CAKE achieves the highest overall score and hypervolume, which indicates a stronger design and broader exploration of the Pareto front. It finds a substantially higher-scoring solution in fewer than 40 trials. The paper reports this result as a tenfold acceleration of the design cycle relative to the baselines.

Photonic-chip optimization curves for score and hypervolume. The black CAKE curve rises fastest and finishes highest in both panels.

Figure 4. Average score and hypervolume over 250 trials for the multi-objective photonic-chip task.

Ablation study

Across all 60 HPOBench tasks, the complete CAKE + BAKER configuration obtains the best average rank, 1.04. Selecting CAKE kernels with acquisition utility alone ranks second at 2.40, while a conventional genetic algorithm4 reaches 2.70 and CAKE with BIC alone reaches 3.02. CKS + BAKER, Adaptive + BAKER, and random recombination rank 3.12, 4.60, and 6.80, respectively. Removing either the LLM-guided evolution or BAKER therefore degrades performance.

ConfigurationAverage rank ↓
CAKE + BAKER1.04
CAKE + Utility2.40
Genetic Algorithm2.70
CAKE + BIC3.02
CKS + BAKER3.12
Adaptive + BAKER4.60
Random Sampling6.80

Table 1. Average rank on HPOBench over 20 random seeds. Lower values are better.

The population analysis helps explain this improvement. After one LLM edit, consisting of one crossover and mutation round, the fitness distribution already shifts toward stronger kernels. After five and ten edits, the mean fitness continues to rise while the variance narrows. Random recombination and traditional genetic operators converge more slowly and retain broader distributions with lower fitness.

Fitness distributions after one, five, and ten edits for LLM, random, and genetic algorithm operators. LLM edits produce the strongest shift toward high fitness.

Figure 5. Kernel-population fitness after 1, 5, and 10 successive edits by the LLM, random recombination, and a genetic algorithm.

Open Question

CAKE adaptively rewrites the kernel during the search, so the surrogate itself changes as data arrive. Classical regret analysis assumes a fixed function class, so deriving bounds when the kernel evolves every round remains open.

Citation

@article{suwandi2025cake,
  title={Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs},
  author={Richard Cornelius Suwandi and Feng Yin and Juntao Wang and Renjie Li and Tsung-Hui Chang and Sergios Theodoridis},
  journal={39th Conference on Neural Information Processing Systems (NeurIPS)},
  year={2025}
}

Footnotes

  1. Weighting by exp(−BIC) turns the criterion into a softmax-style score: kernels with a lower (better) BIC receive exponentially more influence on the final ranking than kernels that fit the data poorly. [↩]
  2. gpt-4o-mini was chosen for its balance of API cost, inference speed, and reasoning ability. We also benchmark CAKE with five other LLMs on HPOBench, including qwen-2.5-7b, llama-3.1-8b, gemini-2.0-flash, claude-3.5-haiku, and the reasoning model deepseek-r1-distill-qwen-7b. Performance improves with more capable LLMs, and the reasoning model gives a further boost at the cost of longer inference time. [↩]
  3. Each CAKE iteration costs about 1.7 seconds for the LLM call plus 3 to 5 seconds for fitting the population's GPs. Since evaluating the black-box objective is usually far more expensive than this overhead, sample efficiency matters more than wall-clock time per iteration. [↩]

References

  1. Gaussian Processes for Machine Learning [PDF]
    Rasmussen, C.E. and Williams, C.K.I., 2006. MIT Press.
  2. Structure Discovery in Nonparametric Regression through Compositional Kernel Search [PDF]
    Duvenaud, D., Lloyd, J., Grosse, R., Tenenbaum, J. and Ghahramani, Z., 2013. International Conference on Machine Learning.
  3. Estimating the Dimension of a Model
    Schwarz, G., 1978. The Annals of Statistics. DOI: 10.1214/aos/1176344136
  4. Adaptation in Natural and Artificial Systems
    Holland, J.H., 1992. MIT Press.
  5. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization [PDF]
    Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A.G. and Bakshy, E., 2020. Advances in Neural Information Processing Systems.
  6. Efficient Global Optimization of Expensive Black-Box Functions
    Jones, D.R., Schonlau, M. and Welch, W.J., 1998. Journal of Global Optimization. DOI: 10.1023/A:1008306431147
  7. Automating Bayesian Optimization with Bayesian Optimization [PDF]
    Malkomes, G. and Garnett, R., 2018. Advances in Neural Information Processing Systems.
  8. HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO [PDF]
    Eggensperger, K., Müller, P., Mallik, N., Feurer, M., Sass, R., Klein, A., Awad, N., Lindauer, M. and Hutter, F., 2021. NeurIPS Datasets and Benchmarks Track.
  9. Batched Large-scale Bayesian Optimization in High-dimensional Spaces [PDF]
    Wang, Z., Gehring, C., Kohli, P. and Jegelka, S., 2018. International Conference on Artificial Intelligence and Statistics.
  10. Gymnasium: A Standard Interface for Reinforcement Learning Environments [link]
    Towers, M. et al., 2024. arXiv:2407.17032.
  11. Deep Reinforcement Learning Empowers Automated Inverse Design and Optimization of Photonic Crystals for Nanoscale Laser Cavities
    Li, R., Zhang, C., Xie, W., Gong, Y., Ding, F., Dai, H., Chen, Z., Yin, F. and Zhang, Z., 2023. Nanophotonics. DOI: 10.1515/nanoph-2022-0693
  12. Multi-objective Bayesian Global Optimization using Expected Hypervolume Improvement Gradient
    Yang, K., Emmerich, M., Deutz, A. and Bäck, T., 2019. Swarm and Evolutionary Computation. DOI: 10.1016/j.swevo.2018.10.007