Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory. (arXiv:2201.02664v3 [cs.LG] UPDATED)

A significant bottleneck in federated learning (FL) is the network
communication cost of sending model updates from client devices to the central
server. We present a comprehensive empirical study of the statistics of model
updates in FL, as well as the role and benefits of various compression
techniques. Motivated by these observations, we propose a novel method to
reduce the average communication cost, which is near-optimal in many use cases,
and outperforms Top-K, DRIVE, 3LC and QSGD on Stack Overflow …

accuracy arxiv communication federated learning learning rate theory trade