Proving Linear Mode Connectivity of Neural Networks via Optimal Transport

arXiv:2310.19103v2 Announce Type: replace
Abstract: The energy landscape of high-dimensional non-convex optimization problems is crucial to understanding the effectiveness of modern deep neural network architectures. Recent works have experimentally shown that two different solutions found after two runs of a stochastic training are often connected by very simple continuous paths (e.g., linear) modulo a permutation of the weights. In this paper, we provide a framework theoretically explaining this empirical observation. Based on convergence rates in Wasserstein distance of empirical measures, …

abstract architectures arxiv connectivity continuous cs.lg deep neural network energy found landscape linear modern network networks neural network neural networks optimization simple solutions stochastic training transport type understanding via