BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling

arXiv:2310.10879v2 Announce Type: replace
Abstract: The increasing complexity of modern deep neural network models and the expanding sizes of datasets necessitate the development of optimized and scalable training methods. In this white paper, we addressed the challenge of efficiently training neural network models using sequences of varying sizes. To address this challenge, we propose a novel training scheme that enables efficient distributed data-parallel training on sequences of different sizes with minimal overhead. By using this scheme we were able to …

abstract arxiv challenge complexity cs.dc cs.lg data datasets deep neural network development modern network network training neural network paper scalable training type white paper