Vanishing Variance Problem in Fully Decentralized Neural-Network Systems

arXiv:2404.04616v1 Announce Type: new
Abstract: Federated learning and gossip learning are emerging methodologies designed to mitigate data privacy concerns by retaining training data on client devices and exclusively sharing locally-trained machine learning (ML) models with others. The primary distinction between the two lies in their approach to model aggregation: federated learning employs a centralized parameter server, whereas gossip learning adopts a fully decentralized mechanism, enabling direct model exchanges among nodes. This decentralized nature often positions gossip learning as less efficient …

abstract aggregation arxiv client concerns cs.dc cs.lg data data privacy decentralized devices federated learning lies machine machine learning network privacy systems training training data type variance