Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training

arXiv:2007.13985v2 Announce Type: replace-cross
Abstract: Stochastic gradient descent~(SGD) and its variants have been the dominating optimization methods in machine learning. Compared to SGD with small-batch training, SGD with large-batch training can better utilize the computational power of current multi-core systems such as graphics processing units~(GPUs) and can reduce the number of communication rounds in distributed training settings. Thus, SGD with large-batch training has attracted considerable attention. However, existing empirical results showed that large-batch training typically leads to a drop in …

abstract arxiv computational core cs.lg current gpus gradient graphics graphics processing units machine machine learning optimization power processing reduce small stat.ml stochastic systems training type units variants