In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

arXiv:2404.16795v1 Announce Type: new
Abstract: With the increasing computational costs associated with deep learning, automated hyperparameter optimization methods, strongly relying on black-box Bayesian optimization (BO), face limitations. Freeze-thaw BO offers a promising grey-box alternative, strategically allocating scarce resources incrementally to different configurations. However, the frequent surrogate model updates inherent to this approach pose challenges for existing methods, requiring retraining or fine-tuning their neural network surrogates online, introducing overhead, instability, and hyper-hyperparameters. In this work, we propose FT-PFN, a novel surrogate …

abstract alternative arxiv automated bayesian box computational context costs cs.lg deep learning face however hyperparameter limitations optimization resources type updates