Topology-aware Generalization of Decentralized SGD. (arXiv:2206.12680v2 [cs.LG] UPDATED)

This paper studies the algorithmic stability and generalizability of
decentralized stochastic gradient descent (D-SGD). We prove that the consensus
model learned by D-SGD is $\mathcal{O}{(m/N+1/m+\lambda^2)}$-stable in
expectation in the non-convex non-smooth setting, where $N$ is the total sample
size of the whole system, $m$ is the worker number, and $1-\lambda$ is the
spectral gap that measures the connectivity of the communication topology.
These results then deliver an
$\mathcal{O}{(1/N+{({(m^{-1}\lambda^2)}^{\frac{\alpha}{2}}+
m^{-\alpha})}/{N^{1-\frac{\alpha}{2}}})}$ in-average generalization bound,
which is non-vacuous even when $\lambda$ is closed …

arxiv decentralized lg topology