Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD. (arXiv:2105.08023v2 [math.OC] UPDATED)

We consider the decentralized stochastic optimization problems, where a
network of $n$ nodes, each owning a local cost function, cooperate to find a
minimizer of the globally-averaged cost. A widely studied decentralized
algorithm for this problem is decentralized SGD (D-SGD), in which each node
averages only with its neighbors. D-SGD is efficient in single-iteration
communication, but it is very sensitive to the network topology. For smooth
objective functions, the transient stage (which measures the number of
iterations the algorithm has …

arxiv data decentralized influence math network