A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks. (arXiv:2210.16286v1 [cs.LG])

To understand the training dynamics of neural networks (NNs), prior studies
have considered the infinite-width mean-field (MF) limit of two-layer NN,
establishing theoretical guarantees of its convergence under gradient flow
training as well as its approximation and generalization capabilities. In this
work, we study the infinite-width limit of a type of three-layer NN model whose
first layer is random and fixed. To define the limiting model rigorously, we
generalize the MF theory of two-layer NNs by treating the neurons as …

arxiv mean networks neural networks space theory