Efficient Non-Parametric Optimizer Search for Diverse Tasks. (arXiv:2209.13575v1 [cs.LG])

Efficient and automated design of optimizers plays a crucial role in
full-stack AutoML systems. However, prior methods in optimizer search are often
limited by their scalability, generability, or sample efficiency. With the goal
of democratizing research and application of optimizer search, we present the
first efficient, scalable and generalizable framework that can directly search
on the tasks of interest. We first observe that optimizer updates are
fundamentally mathematical expressions applied to the gradient. Inspired by the
innate tree structure of …

arxiv non-parametric parametric search