Tune without Validation: Searching for Learning Rate and Weight Decay on Training Sets

arXiv:2403.05532v1 Announce Type: new
Abstract: We introduce Tune without Validation (Twin), a pipeline for tuning learning rate and weight decay without validation sets. We leverage a recent theoretical framework concerning learning phases in hypothesis space to devise a heuristic that predicts what hyper-parameter (HP) combinations yield better generalization. Twin performs a grid search of trials according to an early-/non-early-stopping scheduler and then segments the region that provides the best results in terms of training loss. Among these trials, the weight …

abstract arxiv cs.cv cs.lg framework hypothesis pipeline rate searching space training twin type validation