Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks

arXiv:2403.08837v1 Announce Type: new
Abstract: Training large deep learning models requires parallelization techniques to scale. In existing methods such as Data Parallelism or ZeRO-DP, micro-batches of data are processed in parallel, which creates two drawbacks: the total memory required to store the model's activations peaks at the end of the forward pass, and gradients must be simultaneously averaged at the end of the backpropagation step. We propose Cyclic Data Parallelism, a novel paradigm shifting the execution of the micro-batches from …

abstract arxiv cs.ai cs.dc cs.lg cs.ne data deep learning memory micro networks neural networks parallelization scale stat.ml store the end total training type