Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time

In this paper, we study the optimality gap between two-layer ReLU networks regularized with weight decay and their convex relaxations. We show that when the training data is random, the relative optimality gap between the original problem and its relaxation can be bounded by a factor of $O(\sqrt{\log n})$, where $n$ is the number of training samples. A simple application leads to a tractable polynomial-time algorithm that is guaranteed to solve the original non-convex problem up to a logarithmic factor. …

cs.lg data gap global layer math.oc networks neural networks paper polynomial random relu show study training training data