Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

arXiv:2312.05181v2 Announce Type: replace-cross
Abstract: Deep learning (DL) jobs use multi-dimensional parallelism, i.e. combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. …

abstract arxiv cs.ai cs.dc cs.lg data deep learning dynamic elasticity experience gpu gpus hardware jobs maintenance pipeline running tensor training type