A Codesign of Scheduling and Parallelization for Large Model Training in Heterogeneous Clusters

arXiv:2403.16125v1 Announce Type: cross
Abstract: Joint consideration of scheduling and adaptive parallelism offers great opportunities for improving the training efficiency of large models on heterogeneous GPU clusters. However, integrating adaptive parallelism into a cluster scheduler expands the cluster scheduling space. The new space is the product of the original scheduling space and the parallelism exploration space of adaptive parallelism (also a product of pipeline, data, and tensor parallelism). The exponentially enlarged scheduling space and ever-changing optimal parallelism plan from adaptive …

abstract arxiv cluster cs.dc cs.lg efficiency gpu however improving large models opportunities parallelization product scheduling space training type