DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updates

arXiv:2307.07652v2 Announce Type: replace
Abstract: Two widely considered decentralized learning algorithms are Gossip and random walk-based learning. Gossip algorithms (both synchronous and asynchronous versions) suffer from high communication cost, while random-walk based learning experiences increased convergence time. In this paper, we design a fast and communication-efficient asynchronous decentralized learning mechanism DIGEST by taking advantage of both Gossip and random-walk ideas, and focusing on stochastic gradient descent (SGD). DIGEST is an asynchronous decentralized algorithm building on local-SGD algorithms, which are originally …

abstract algorithms arxiv asynchronous communication convergence cost cs.dc cs.lg decentralized design paper random random-walk replace type updates versions while