Layer-wise Linear Mode Connectivity

arXiv:2307.06966v3 Announce Type: replace
Abstract: Averaging neural network parameters is an intuitive method for fusing the knowledge of two independent models. It is most prominently used in federated learning. If models are averaged at the end of training, this can only lead to a good performing model if the loss surface of interest is very particular, i.e., the loss in the midpoint between the two models needs to be sufficiently low. This is impossible to guarantee for the non-convex losses …

abstract arxiv connectivity cs.lg federated learning good independent knowledge layer linear loss network neural network parameters surface the end training type wise