Hydra: A System for Large Multi-Model Deep Learning. (arXiv:2110.08633v7 [cs.DC] UPDATED)

Scaling up model depth and size is now a common approach to raise accuracy in
many deep learning (DL) applications, as evidenced by the widespread success of
multi-billion or even trillion parameter models in natural language processing
(NLP) research. Despite success in DL research and at major technology
companies, broader practical adoption of such large models among domain
scientists and businesses is still bottlenecked by GPU memory limits, high
training costs, and low GPU availability, even on public clouds. Model …

arxiv deep learning hydra learning