Scrap Your Schedules with PopDescent

arXiv:2310.14671v2 Announce Type: replace
Abstract: In contemporary machine learning workloads, numerous hyper-parameter search algorithms are frequently utilized to efficiently discover high-performing hyper-parameter values, such as learning and regularization rates. As a result, a range of parameter schedules have been designed to leverage the capability of adjusting hyper-parameters during training to enhance loss performance. These schedules, however, introduce new hyper-parameters to be searched and do not account for the current loss values of the models being trained.
To address these issues, …

abstract adjusting algorithms arxiv capability cs.lg loss machine machine learning parameters performance regularization search search algorithms training type values workloads