Asynchronous Training Schemes in Distributed Learning with Time Delay. (arXiv:2208.13154v1 [cs.LG])

In the context of distributed deep learning, the issue of stale weights or
gradients could result in poor algorithmic performance. This issue is usually
tackled by delay tolerant algorithms with some mild assumptions on the
objective functions and step sizes. In this paper, we propose a different
approach to develop a new algorithm, called $\textbf{P}$redicting
$\textbf{C}$lipping $\textbf{A}$synchronous $\textbf{S}$tochastic
$\textbf{G}$radient $\textbf{D}$escent (aka, PC-ASGD). Specifically, PC-ASGD
has two steps - the $\textit{predicting step}$ leverages the gradient
prediction using Taylor expansion to reduce …

arxiv asynchronous distributed learning time training