Efficient Strong Scaling Through Burst Parallel Training. (arXiv:2112.10065v3 [cs.DC] UPDATED)

As emerging deep neural network (DNN) models continue to grow in size, using
large GPU clusters to train DNNs is becoming an essential requirement to
achieving acceptable training times. In this paper, we consider the case where
future increases in cluster size will cause the global batch size that can be
used to train models to reach a fundamental limit: beyond a certain point,
larger global batch sizes cause sample efficiency to degrade, increasing
overall time to accuracy. As a …

arxiv scaling training