Do Deep Neural Network Solutions Form a Star Domain?

arXiv:2403.07968v1 Announce Type: new
Abstract: Entezari et al. (2022) conjectured that neural network solution sets reachable via stochastic gradient descent (SGD) are convex, considering permutation invariances. This means that two independent solutions can be connected by a linear path with low loss, given one of them is appropriately permuted. However, current methods to test this theory often fail to eliminate loss barriers between two independent solutions (Ainsworth et al., 2022; Benzing et al., 2022). In this work, we conjecture that …

arxiv cs.ai cs.lg deep neural network domain form network neural network solutions star type