Tesseract: Parallelize the Tensor Parallelism Efficiently. (arXiv:2105.14500v2 [cs.DC] UPDATED)

Together with the improvements in state-of-the-art accuracies of various
tasks, deep learning models are getting significantly larger. However, it is
extremely difficult to implement these large models because limited GPU memory
makes it impossible to fit large models into a single GPU or even a GPU server.
Besides, it is highly necessary to reduce the training time for large models.
Previous methods like Megatron-LM implemented a 1-Dimensional distributed
method to use GPUs to speed up the training. However, these methods …

arxiv tensor tesseract