Principled Architecture-aware Scaling of Hyperparameters

arXiv:2402.17440v1 Announce Type: new
Abstract: Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, we precisely characterize the dependence of initializations and maximal learning …

abstract architecture arxiv cs.lg current deep neural network design designs diverse network neural network optimization process quality scaling training type