Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm

arXiv:2306.02939v2 Announce Type: replace
Abstract: This paper presents a new generalization error analysis for Decentralized Stochastic Gradient Descent (D-SGD) based on algorithmic stability. The obtained results overhaul a series of recent works that suggested an increased instability due to decentralization and a detrimental impact of poorly-connected communication graphs on generalization. On the contrary, we show, for convex, strongly convex and non-convex functions, that D-SGD can always recover generalization bounds analogous to those of classical SGD, suggesting that the choice of …

abstract algorithm analysis arxiv communication cs.lg decentralization decentralized error gradient graphs impact paper series stability stat.ml stochastic type