Random ReLU Neural Networks as Non-Gaussian Processes

arXiv:2405.10229v1 Announce Type: cross
Abstract: We consider a large class of shallow neural networks with randomly initialized parameters and rectified linear unit activation functions. We prove that these random neural networks are well-defined non-Gaussian processes. As a by-product, we demonstrate that these networks are solutions to stochastic differential equations driven by impulsive white noise (combinations of random Dirac measures). These processes are parameterized by the law of the weights and biases as well as the density of activation thresholds in …

abstract arxiv class cs.lg differential functions gaussian processes linear math.pr networks neural networks parameters processes product prove random relu solutions stat.ml stochastic type