Smart Distributed Training on Amazon SageMaker with SMD: Part 2

How to Optimize Data Distribution with SageMaker Distributed Data Parallel

This is the second part of a three-part post on the topic of optimizing distributed training. In part one, we provided a brief survey of distributed training algorithms. We noted that common to all algorithms is their reliance on high-speed communication between multiple GPUs. We surmised that a distributed algorithm that accounted for the underlying instance topology, particularly the differences in the communication links …

amazon amazon sagemaker distributed distributed-training machine learning part sagemaker smart tensorflow training