Partitioned Neural Network Training via Synthetic Intermediate Labels

arXiv:2403.11204v1 Announce Type: new
Abstract: The proliferation of extensive neural network architectures, particularly deep learning models, presents a challenge in terms of resource-intensive training. GPU memory constraints have become a notable bottleneck in training such sizable models. Existing strategies, including data parallelism, model parallelism, pipeline parallelism, and fully sharded data parallelism, offer partial solutions. Model parallelism, in particular, enables the distribution of the entire model across multiple GPUs, yet the ensuing data communication between these partitions slows down training. Additionally, …

abstract architectures arxiv become challenge constraints cs.ai cs.dc cs.lg data deep learning gpu intermediate labels memory network network training neural network pipeline strategies synthetic terms training type via