Alpa: Automating Model Sharding for Distributed Deep Learning

A new open-source library called Alpa aims to automate distributed training and serving of large deep networks. It proposes a compiler where existing model-parallel strategies are combined and the usage of computing resources is optimized according to the deep network architecture.

ai data science deep learning distributed machine learning ml & data engineering news sharding