Implicit regularization of deep residual networks towards neural ODEs

arXiv:2309.01213v2 Announce Type: replace-cross
Abstract: Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous models still lacks a solid mathematical foundation. In this article, we take a step in this direction by establishing an implicit regularization of deep residual networks towards neural ODEs, for nonlinear networks trained with gradient flow. We prove that if the network is initialized as …

abstract analog art article arxiv continuous cs.lg deep learning differential foundation networks neural networks ordinary regularization residual solid state stat.ml success type