Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks

arXiv:2310.12079v2 Announce Type: replace-cross
Abstract: Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential equations. However, these results do not a priori tell us anything about "ordinary" unshaped networks, where the activation is unchanged as the network size grows. In this article, we find similar differential equation based asymptotic characterization for two types of unshaped networks.
Firstly, we show that the following …

abstract arxiv cs.lg differential differential equation equation function however network networks neural networks ordinary results scaling stat.ml type