Generalization of Scaled Deep ResNets in the Mean-Field Regime

arXiv:2403.09889v1 Announce Type: new
Abstract: Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate \emph{scaled} ResNet in the limit of infinitely deep and wide neural networks, of which the gradient flow is described by a partial differential equation in the large-neural network limit, i.e., the \emph{mean-field} regime. To derive the generalization bounds under this setting, our analysis necessitates a shift from the conventional …

abstract arxiv beyond cs.lg flow gradient lazy mean networks neural networks resnet success training type work