Towards Learning Universal Hyperparameter Optimizers with Transformers. (arXiv:2205.13320v1 [cs.LG])

Meta-learning hyperparameter optimization (HPO) algorithms from prior
experiments is a promising approach to improve optimization efficiency over
objective functions from a similar distribution. However, existing methods are
restricted to learning from experiments sharing the same set of
hyperparameters. In this paper, we introduce the OptFormer, the first
text-based Transformer HPO framework that provides a universal end-to-end
interface for jointly learning policy and function prediction when trained on
vast tuning data from the wild. Our extensive experiments demonstrate that the
OptFormer …

arxiv learning transformers