On the Acceleration of Deep Learning Model Parallelism with Staleness. (arXiv:1909.02625v3 [cs.LG] UPDATED)

Training the deep convolutional neural network for computer vision problems
is slow and inefficient, especially when it is large and distributed across
multiple devices. The inefficiency is caused by the backpropagation algorithm's
forward locking, backward locking, and update locking problems. Existing
solutions for acceleration either can only handle one locking problem or lead
to severe accuracy loss or memory inefficiency. Moreover, none of them consider
the straggler problem among devices. In this paper, we propose Layer-wise
Staleness and a novel …

arxiv deep learning learning