Better Schedules for Low Precision Training of Deep Neural Networks

arXiv:2403.02243v1 Announce Type: new
Abstract: Low precision training can significantly reduce the computational overhead of training deep neural networks (DNNs). Though many such techniques exist, cyclic precision training (CPT), which dynamically adjusts precision throughout train- ing according to a cyclic schedule, achieves particularly impressive improvements in training efficiency, while actually improving DNN performance. Existing CPT implementations take common learning rate schedules (e.g., cyclical cosine sched- ules) and use them for low precision training without adequate comparisons to alternative scheduling options. …

abstract arxiv computational cs.ai cs.lg efficiency improvements ing low networks neural networks precision reduce train training type