Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport

arXiv:2403.12887v1 Announce Type: new
Abstract: We study the convergence of gradient flow for the training of deep neural networks. If Residual Neural Networks are a popular example of very deep architectures, their training constitutes a challenging optimization problem due notably to the non-convexity and the non-coercivity of the objective. Yet, in applications, those tasks are successfully solved by simple optimization algorithms such as gradient descent. To better understand this phenomenon, we focus here on a ``mean-field'' model of infinitely deep …

abstract architectures arxiv convergence cs.lg example flow gradient math.oc networks neural networks optimization popular residual study training transport type understanding