Simultaneous linear connectivity of neural networks modulo permutation

arXiv:2404.06498v1 Announce Type: new
Abstract: Neural networks typically exhibit permutation symmetries which contribute to the non-convexity of the networks' loss landscapes, since linearly interpolating between two permuted versions of a trained network tends to encounter a high loss barrier. Recent work has argued that permutation symmetries are the only sources of non-convexity, meaning there are essentially no such barriers between trained networks if they are permuted appropriately. In this work, we refine these arguments into three distinct claims of increasing …

abstract arxiv connectivity cs.lg linear loss network networks neural networks stat.ml type versions work