Designing Stable Neural Networks using Convex Analysis and ODEs

arXiv:2306.17332v2 Announce Type: replace
Abstract: Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further …

abstract analysis architecture arxiv cs.lg designing differential integration network network architecture networks neural network neural networks numerical operators ordinary resnet type work