Parallel Deep Neural Networks Have Zero Duality Gap. (arXiv:2110.06482v2 [cs.LG] UPDATED)

Training deep neural networks is a well-known highly non-convex problem. In
recent works, it is shown that there is no duality gap for regularized
two-layer neural networks with ReLU activation, which enables global
optimization via convex programs. For multi-layer linear networks with vector
outputs, we formulate convex dual problems and demonstrate that the duality gap
is non-zero for depth three and deeper networks. However, by modifying the deep
networks to more powerful parallel architectures, we show that the duality gap …

arxiv gap lg networks neural networks