Hidden Minima in Two-Layer ReLU Networks

arXiv:2312.16819v2 Announce Type: replace
Abstract: The optimization problem associated to fitting two-layer ReLU networks having $d$~inputs, $k$~neurons, and labels generated by a target network, is considered. Two types of infinite families of spurious minima, giving one minimum per $d$, were recently found. The loss at minima belonging to the first type converges to zero as $d$ increases. In the second type, the loss remains bounded away from zero. That being so, how may one avoid minima belonging to the latter …

abstract arxiv cs.lg families found generated giving hidden inputs labels layer loss math.oc network networks neurons optimization per relu stat.ml type types