Near-Optimal Scaling of Large Deep Network Training on Public Cloud

A recently published study, MiCS, provides experimental evidence that the infrastructure used to carry out model training should be taken into account, especially for large deep neural networks trained on the public cloud. The article shows distributing the model weights unevenly between GPUs decreases inter-node communication overhead on AWS V100 and A100 instances.

ai aws cloud cloud computing deep learning ml & data engineering near network network training news public public cloud scaling training