OneFlow: Redesign the Distributed Deep Learning Framework from Scratch. (arXiv:2110.15032v3 [cs.DC] UPDATED)

Deep learning frameworks such as TensorFlow and PyTorch provide a productive
interface for expressing and training a deep neural network (DNN) model on a
single device or using data parallelism. Still, they may not be flexible or
efficient enough in training emerging large models on distributed devices,
which require more sophisticated parallelism beyond data parallelism. Plugins
or wrappers have been developed to strengthen these frameworks for model or
pipeline parallelism, but they complicate the usage and implementation of
distributed deep …

arxiv deep learning deep learning framework distributed framework learning