Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarks

arXiv:2403.01888v1 Announce Type: cross
Abstract: While deep learning has celebrated many successes, its results often hinge on the meticulous selection of hyperparameters (HPs). However, the time-consuming nature of deep learning training makes HP optimization (HPO) a costly endeavor, slowing down the development of efficient HPO tools. While zero-cost benchmarks, which provide performance and runtime without actual training, offer a solution for non-parallel setups, they fall short in parallel setups as each worker must communicate its queried runtime to return its …

abstract arxiv asynchronous benchmarking benchmarks cost cs.ai cs.lg deep learning deep learning training development endeavor fidelity hinge nature optimization results tools training type