Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs. (arXiv:2110.05518v2 [cs.LG] UPDATED)

Understanding the fundamental mechanism behind the success of deep neural
networks is one of the key challenges in the modern machine learning
literature. Despite numerous attempts, a solid theoretical analysis is yet to
be developed. In this paper, we develop a novel unified framework to reveal a
hidden regularization mechanism through the lens of convex optimization. We
first show that the training of multiple three-layer ReLU sub-networks with
weight decay regularization can be equivalently cast as a convex optimization
problem …

arxiv global networks training