On Distributed Adaptive Optimization with Gradient Compression. (arXiv:2205.05632v1 [stat.ML])

We study COMP-AMS, a distributed optimization framework based on gradient
averaging and adaptive AMSGrad algorithm. Gradient compression with error
feedback is applied to reduce the communication cost in the gradient
transmission process. Our convergence analysis of COMP-AMS shows that such
compressed gradient averaging strategy yields same convergence rate as standard
AMSGrad, and also exhibits the linear speedup effect w.r.t. the number of local
workers. Compared with recently proposed protocols on distributed adaptive
methods, COMP-AMS is simple and convenient. Numerical experiments …

arxiv compression distributed gradient ml optimization