Preserved central model for faster bidirectional compression in distributed settings. (arXiv:2102.12528v2 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2102.12528

We develop a new approach to tackle communication constraints in a
distributed learning problem with a central server. We propose and analyze a
new algorithm that performs bidirectional compression and achieves the same
convergence rate as algorithms using only uplink (from the local workers to the
central server) compression. To obtain this improvement, we design MCM, an
algorithm such that the downlink compression only impacts local models, while
the global model is preserved. As a result, and contrary to previous …

arxiv compression distributed lg model