DiPaCo: Distributed Path Composition

arXiv:2403.10616v1 Announce Type: new
Abstract: Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high bandwidth communication between devices working in parallel. In this work, we propose a co-designed modular architecture and training approach for ML models, dubbed DIstributed PAth COmposition (DiPaCo). During training, DiPaCo distributes computation by paths through a set of shared modules. Together with …

abstract architecture arxiv bandwidth communication cs.cl cs.lg devices distributed engineering machine machine learning modular network neural network path progress scaling training type work