Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters. (arXiv:2205.09470v1 [cs.LG])

The ever-growing model size and scale of compute have attracted increasing
interests in training deep learning models over multiple nodes. However, when
it comes to training on cloud clusters, especially across remote clusters, huge
challenges are faced. In this work, we introduce a general framework, Nebula-I,
for collaboratively training deep learning models over remote heterogeneous
clusters, the connections between which are low-bandwidth wide area networks
(WANs). We took natural language processing (NLP) as an example to show how
Nebula-I works …

arxiv cloud deep learning framework general learning training