Efficient Global Optimization of Two-layer ReLU Networks: Quadratic-time Algorithms and Adversarial Training. (arXiv:2201.01965v1 [cs.LG])

The non-convexity of the artificial neural network (ANN) training landscape
brings inherent optimization difficulties. While the traditional
back-propagation stochastic gradient descent (SGD) algorithm and its variants
are effective in certain cases, they can become stuck at spurious local minima
and are sensitive to initializations and hyperparameters. Recent work has shown
that the training of an ANN with ReLU activations can be reformulated as a
convex program, bringing hope to globally optimizing interpretable ANNs.
However, naively solving the convex training formulation …

algorithms arxiv global networks optimization time training